AIS.NET Projects

Package	Status
Ais.Net
Ais.Net.Models
Ais.Net.Receiver
Ais.Net.Receiver.Storage.Azure.Blob
IP Maturity Model Score
Build Status

The AIS.NET project contains a series of layers, from a low-level high performance NMEA AIS sentence decoder, to a rich high-level C# 9.0 models of AIS message types, a receiver component that can listen to TCP streams of NMEA sentences and expose them as an IObservable<string> of raw sentences or an decoded IObservable<IAisMessage>, and finally a Storage Client implementation to persisting the raw NMEA sentence stream to Azure Blob storage for future processing.

Ais.Net.Receiver

A simple .NET 5.0 AIS Receiver for capturing the Norwegian Coastal Administration's marine Automatic Identification System (AIS) AIVDM/AIVDO NMEA message network data (available under Norwegian license for public data (NLOD)) and persisting in Microsoft Azure Blob Storage.

The Norwegian Costal Administration provide a TCP endpoint (153.44.253.27:5631) for broadcasting their raw AIS AIVDM/AIVDO sentences, captured by over 50 base stations, and covers the area 40-60 nautical miles from the Norwegian coastline.

This project contains a NmeaReceiver which consumes the raw NetworkStream, a NmeaToAisMessageTypeProcessor, which can decode the raw sentences into IAisMessage, and ReceiverHost which manages the process and exposes an IObservable<string> for raw sentences and an IObservable<IAisMessage> for decoded messages. ReceiverHost can be hosted in a console application or other runtime environments like .NET Interactive.

The project also includes a demo console which shows how the various pieces can fit together, including subscribing to the IObservable<string> and IObservable<IAisMessage> streams and displaying the results or batch the AIVDM/AIVDO sentences and write them to Azure Blob Storage using the Append Blob feature, to create timestamped hour-long rolling logs.

The purpose of this application is to provide sample data for Ais.Net - the .NET Standard, high performance, zero allocation AIS decoder. The majority of raw AIS data is only available via commercial sources, and thus creating AIS datasets large enough to test / benchmark Ais.Net is almost impossible.

The Norwegian Costal Administration TCP endpoint produces:

• ~2.9 KB per second
• ~10.3 MB per hour
• ~248 MB per day
• ~1.7 GB per week
• ~7 GB per month
• ~81.4 GB per year

Ais.Net.Models

Ais.Net.Receiver bridges the gap between the high performance, zero allocation world of Ais.Net and the real world need for types to perform meaningful operations. Ais.Net.Models provides a series of C# 9.0 records which define the the message types, a series of interfaces that define common behaviours, and extension methods to help with type conversions & calculations.

The table below shows the messages, their properties and how they are mapped to interfaces.

Show AIS Message Types and .NET Interfaces

	Message Type 1 to 3	Message Type 5	Message Type 18	Message Type 19	Message Type 24 Part 0	Message Type 24 Part 1
IAisMessageType5		AisVersion
ICallSign		CallSign				CallSign
IAisMessageType18			CanAcceptMessage22ChannelAssignment
IAisMessageType18			CanSwitchBands
IVesselNavigation	CourseOverGround10thDegrees		CourseOverGround10thDegrees	CourseOverGround10thDegrees
IAisMessageType5		Destination
IAisMessageType18			CsUnit
IVesselDimensions		DimensionToBow		DimensionToBow		DimensionToBow
IVesselDimensions		DimensionToPort		DimensionToPort		DimensionToPort
IVesselDimensions		DimensionToStarboard		DimensionToStarboard		DimensionToStarboard
IVesselDimensions		DimensionToStern		DimensionToStern		DimensionToStern
IAisMessageType5		Draught10thMetres
IAisMessageType5		EtaMonth
IAisMessageType5		EtaDay
IAisMessageType5		EtaHour
IAisMessageType5		EtaMinute
IAisMessageType18			HasDisplay
IIsAssigned			IsAssigned	IsAssigned
IAisMessageType18			IsDscAttached
IAisMessageType5		ImoNumber
IAisIsDteNotReady		IsDteNotReady		IsDteNotReady
IVesselNavigation	Latitude10000thMins		Latitude10000thMins	Latitude10000thMins
IVesselNavigation	Longitude10000thMins		Longitude10000thMins	Longitude10000thMins
IAisMessageType1to3	ManoeuvreIndicator
IAisMessageType24Part1						MothershipMmsi
IAisMessageType	MessageType	MessageType	MessageType	MessageType	MessageType	MessageType
IVesselIdentity	Mmsi	Mmsi	Mmsi	Mmsi	Mmsi	Mmsi
IAisMessageType1to3	NavigationStatus
IAisMultipartMessage					PartNumber	PartNumber
IVesselNavigation	PositionAccuracy		PositionAccuracy	PositionAccuracy
IAisPositionFixType		PositionFixType		PositionFixType
IAisMessageType18			RadioStatusType
IAisMessageType1to3	RadioSlotTimeout
IAisMessageType1to3	RadioSubMessage
IAisMessageType1to3	RadioSyncState
IAisMessageType19				RegionalReserved139
IAisMessageType19				RegionalReserved38
IRaimFlag	RaimFlag		RaimFlag	RaimFlag
IAisMessageType18			RegionalReserved139
IAisMessageType18			RegionalReserved38
IAisMessageType1to3	RateOfTurn
IRepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator
IAisMessageType24Part1						SerialNumber
IAisMessageType19				ShipName
IShipType		ShipType		ShipType		ShipType
IAisMessageType1to3	SpareBits145
IAisMessageType24Part0					Spare160
IAisMessageType24Part1						Spare162
IAisMessageType5		Spare423
IAisMessageType19				Spare308
IVesselNavigation	SpeedOverGroundTenths		SpeedOverGroundTenths	SpeedOverGroundTenths
IVesselNavigation	TimeStampSecond		TimeStampSecond	TimeStampSecond
IVesselNavigation	TrueHeadingDegrees		TrueHeadingDegrees	TrueHeadingDegrees
IAisMessageType24Part1						UnitModelCode
IAisMessageType24Part1						VendorIdRev3
IAisMessageType24Part1						VendorIdRev4
IVesselName		VesselName

The C# record types then implement the relevant interfaces, which enables simpler higher level programming constructs, such as Rx queries over an IAisMessage stream:

IObservable<IGroupedObservable<uint, IAisMessage>> byVessel = receiverHost.Messages.GroupBy(m => m.Mmsi);

IObservable<(uint mmsi, IVesselNavigation navigation, IVesselName name)>? vesselNavigationWithNameStream =
    from perVesselMessages in byVessel
    let vesselNavigationUpdates = perVesselMessages.OfType<IVesselNavigation>()
    let vesselNames = perVesselMessages.OfType<IVesselName>()
    let vesselLocationsWithNames = vesselNavigationUpdates.CombineLatest(vesselNames, (navigation, name) => (navigation, name))
    from vesselLocationAndName in vesselLocationsWithNames
    select (mmsi: perVesselMessages.Key, vesselLocationAndName.navigation, vesselLocationAndName.name);

Azure Blob Storage Taxonomy

The AIS data is stored using the following taxonomy

<USER DEFINED CONTAINER NAME>/raw/yyyy/MM/dd/yyyyMMddTHH.nm4

An example directory listing, with a user defined container name of nmea-ais would look as follows:

\---nmea-ais
    \---raw
        \---2021
            \---07
                +---12
                | 20210712T00.nm4 |
                | 20210712T01.mm4 |
                | 20210712T02.nm4 |
                | 20210712T03.nm4 |
                | 20210712T04.nm4 |
                | 20210712T05.nm4 |
                | 20210712T06.nm4 |
                | 20210712T07.nm4 |
                | 20210712T08.nm4 |
                | 20210712T09.nm4 |
                | 20210712T10.nm4 |
                | 20210712T11.nm4 |
                | 20210712T12.nm4 |
                | 20210712T13.nm4 |
                | 20210712T14.nm4 |
                | 20210712T15.nm4 |
                | 20210712T16.nm4 |
                | 20210712T17.nm4 |
                | 20210712T18.nm4 |
                | 20210712T19.nm4 |
                | 20210712T20.nm4 |
                | 20210712T21.nm4 |
                | 20210712T22.nm4 |
                | 20210712T23.nm4 |
                +---20210713
                | 20210713T00.nm4 |

To Run

Update the values in the settings.json file:

{
  "Ais": {
    "host": "153.44.253.27",
    "port": "5631",
    "retryAttempts": 100,
    "retryPeriodicity": "00:00:00:00.500"
  },
  "Storage": {
    "connectionString": "<YOUR AZURE STORAGE CONNECTION STRING>",
    "containerName": "nmea-ais",
    "writeBatchSize": 500
  }
}

From the command line: dotnet Ais.Net.Receiver.Host.Console.exe

Raspberry Pi

As the AIS.NET stack is written in .NET 6.0 and .NET Standard you can publish the Ais.Net.Recevier.Host.Console application with a target runtime of Portable. This will allow you to run the recevier on a Raspberry Pi if you want to capture your own AIS data.

For reliability you can run Ais.Net.Recevier.Host.Console.dll as daemon.

Installation

The combination of Windows Terminal, .NET and PowerShell make a Raspberry Pi a very productive environment for .NET Devs.

Install Windows Terminal. You can download Windows Terminal from the Microsoft Store or from the GitHub releases page.

Open Windows Terminal and use ssh pi@<Raspberry PI IP Address> to connect to your Pi.

Install .NET 6.0

Use the following commands to install .NET 6.0 on your Pi.

1. curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --channel Current
2. echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc
3. echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc
4. source ~/.bashrc
5. dotnet --version

Install PowerShell 7.x

Use the following commands to install PowerShell on your Pi.

1. Download the latest package wget https://github.com/PowerShell/PowerShell/releases/download/v7.2.7/powershell-7.2.7-linux-arm32.tar.gz
2. Create a directory for it to be unpacked into mkdir ~/powershell
3. Unpack tar -xvf ./powershell-7.2.7-linux-arm32.tar.gz -C ~/powershell
4. Give it executable rights sudo chmod +x /opt/microsoft/powershell/7/pwsh
5. Create a symbolic link sudo ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh

use the command pwsh to enter the PowerShell session.

Install Ais.Net.Receiver.Host.Console

2. From the solution root, open a command prompt and type dotnet publish -c Release .\Solutions\Ais.Net.Receiver.sln
3. Add your Azure Blob Storage Account connection string to settings.json
4. Transfer (I use Beyond Compare as it has native SSH support) the contents of .\Solutions\Ais.Net.Receiver.Host.Console\bin\Release\net5.0\publish to a folder called aisr in the home/pi directory on your Raspberry Pi (assuming you still have the default set up.)
5. Copy Solutions\Ais.Net.Receiver.Host.Console.RaspberryPi\aisr.service to /lib/systemd/system/aisr.service
6. run sudo chmod 644 /lib/systemd/system/aisr.service
7. run sudo systemctl enable aisr.service
8. run sudo reboot

You can use journalctl -u "aisr" to view the console output of Ais.Net.Recevier.Host.Console.dll

You can use sudo systemctl restart aisr to restart the service.

If you need to look at / edit the deployed aisr.service use sudo nano /lib/systemd/system/aisr.service make your edits then use Ctrl+O and Ctrl+X to save the file and exit.

Use Azure Storage Explorer to browse to where files are captured.

Configuration

Configuration is read from settings.json and can also be overridden for local development by using a settings.local.json file.

{
  "Ais": {
    "host": "153.44.253.27",
    "port": "5631",
    "loggerVerbosity": "Minimal", 
    "statisticsPeriodicity": "00:01:00",
    "retryAttempts": 100,
    "retryPeriodicity": "00:00:00:00.500"
  },
  "Storage": {
    "enableCapture": true,
    "connectionString": "DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>",
    "containerName": "nmea-ais-dev",
    "writeBatchSize": 500
  }
}

AIS

These settings control the ReceiverHost and its behaviour.

• host: IP Address or FQDN of the AIS Source
• port: Port number for the AIS Source
• loggerVerbosity: Controls the output to the console.
  • Quiet = Essential only,
  • Minimal = Statistics only. Sample rate of statistics controlled by statisticsPeriodicity,
  • Normal = Vessel Names and Positions,
  • Detailed = NMEA Sentences,
  • Diagnostic = Messages and Errors
• statisticsPeriodicity: TimeSpan defining the sample rate of statistics to display
• retryAttempts: Number of retry attempts when a connection error occurs
• retryPeriodicity: How long to wait before a retry attempt.

Storage

These settings control the capturing NMEA sentences to Azure Blob Storage.

• enableCapture: Whether you want to capture the NMEA sentences and write them to Azure Blob Storage
• connectionString: Azure Storage Account Connection String
• containerName: Name of the container to capture the NMEA sentences. You can use this to separate a local dev storage container from your production storage container, within the same storage account.
• writeBatchSize: How many NMEA sentences to batch before writing to Azure Blob Storage.

Licenses

AIS.Net.Receiver is also available under the Apache 2.0 open source license.

The Data ingested by the AIS.Net.Receiver is licensed under the Norwegian license for public data (NLOD).

Project Sponsor

This project is sponsored by endjin, a UK based Microsoft Gold Partner for Cloud Platform, Data Platform, Data Analytics, DevOps, Power BI Partner, and .NET Foundation Corporate Sponsor.

For more information about our products and services, or for commercial support of this project, please contact us.

We produce two free weekly newsletters; Azure Weekly for all things about the Microsoft Azure Platform, and Power BI Weekly.

Keep up with everything that's going on at endjin via our blog, follow us on Twitter, or LinkedIn.

Our other Open Source projects can be found on GitHub

Code of conduct

This project has adopted a code of conduct adapted from the Contributor Covenant to clarify expected behavior in our community. This code of conduct has been adopted by many other projects. For more information see the Code of Conduct FAQ or contact hello@endjin.com with any additional questions or comments.

IP Maturity Model (IMM)