LOGO SIMPLE.JPG
HOM black.JPG

Helm Order Monitor (patent pending) is the first of its kind technological solution to the problem of insufficient monitoring of helm orders between helmsman and pilot or navigator onboard ships.

 

Did you know that disaster of Costa Concordia happened due to the helmsman applying the helm on the wrong side shortly before the impact?

 

We are fixing this problem with an electronic device that combines automatic speech recognition and data from the ship's sensors. This artificial intelligence allows us to continuously monitor whether issued helm orders are clear, confirmed, and most importantly, correctly executed.

Apart from shipboard use, our product is utilized in Nautical simulators for training and monitoring proper use of helm orders as per IMO Standard Marine Communication Phrases.

To use an analogy with road transportation, Helm Order Monitor is like Lane Departure Warning.

 
Ground 3.JPG
Ground 4.JPG

PROBLEM

Below are the main problems that Helm Order Monitor will solve:

  • Most obvious problem is that nothing prevents a helmsman from putting the rudder over the wrong way. The helmsman can also misinterpret or otherwise wrongly execute issued helm orders. Therefore, the monitoring of the execution of the order remains of paramount importance.

  • Most automated warnings are not available in hand steering, such as Default RoT deviation warning, Drift limit alarm, Max. rudder alarm and Low-speed alarm.

  • Monitoring of helm orders requires a high level of concentration which reduces situational awareness.

A significant increase in a number of screens and functions that the bridge team must monitor makes monitoring of helm orders more difficult than before.

In an investigation report of a marine accident that involved helmsman applying wrong the helm, Australian Transport Safety Bureau stated: THIS TYPE OF ERROR IS NOT UNCOMMON.

As a lesser-known fact, in the last 5 minutes before the Costa Concordia breached the hull near Giglio island, a series of wrong helm order interpretations were recorded on VDR, including the wrong helm at a crucial moment, just 22 seconds before the impact. From the chronological records, we made a video reconstruction of last 5 minutes before the impact to see if the use of Helm Order Monitor could have prevented this casualty.

STATE OF THE ART

At open sea, ships mostly sail with autopilot or trackpilot turned on, but to enter/exit the port or pass-through narrow channels the ship must go to manual steering. The steering is done by the helmsman while a maritime pilot or navigator issues verbal helm orders.

Automatic helm monitoring is only available when the ship has engaged auto-pilot, or trackpilot in either heading, course, or track mode. But when the ship is in the manual mode, also called “hand steering”, monitoring is reduced to Rudder angle indicator, compass, and ROT (rate of turn) indicator.

In other words, if the helmsman applies the wrong helm, there will be no warning. Unless caught by the bridge team, the ship will continue to turn to the wrong side.

Case studies have shown that the average time for a bridge team to notice the wrong helm is 7 to 8 seconds. As per IMO Initial Turning Ability Criterion, with the application of a 10-degree rudder angle, the vessel must not travel more than 2.5 ship lengths before the vessel’s heading has changed 10 degrees. If this is translated in higher rudder angles, it is clear that 7 to 8 seconds of the wrong helm can be impossible to correct. That is why monitoring the order’s execution remains of paramount importance, as documented in ships’ manuals.

 

There will also be no warning if the ship deviates from a steady course. The only way to see this is by constant monitoring of the compass heading or the ground reference.

For orders that include ROT, no warning is given by the ROT indicator if the helmsman exceeds the required ROT. This can lead to an excessive list, which particularly on passenger ships can cause injuries.

LANGUAGE BARRIER

In a study conducted by Transport Canada, when pilots were asked whether language barriers make it difficult to communicate orders to the helmsman on foreign-registered vessels, some 60% replied that language barriers “sometimes” affect communication with the helmsman while 20% reported that it “often” resulted in difficulty in communicating.

Helm Order Monitor would force the helmsman to try and clearly repeat the helm orders, thus reducing the strain on the pilots.

INTERVIEW WITH A CAPTAIN REGARDING PASSAGE THROUGH SUEZ CANAL

I'm on the Navigational bridge all the time (passage lasts 12 to 13 hours).

 

I leave nothing to chance, I follow every pilot’s helm order and helmsman response. Several times, it happened that because of a misunderstanding between the pilot and helmsman, I had to jump and grab the helm to avoid ship grounding.

 

Such a situation can also occur because of a simple misinterpretation, someone hears the wrong figure, and everything goes wrong in the second.

image006.jpg

Foto: MAXAR TECHNOLOGIES

RESEARCH

As there was no feasible solution, helmsman errors were considered an unavoidable risk in the maritime industry. The number of close call situations that could end in loss of lives and oil pollution counts in hundreds on a yearly base and we can only guess how many accidents occurred by the fatigue contributed by these obsolete procedures.

While researching into the number of accidents caused by helmsman error, we realized that these errors are not classified. We searched each country's investigative reports for marine accidents such as collisions and groundings and found that helmsman errors are quite a common occurrence. We are continuing our research as many countries are not making their investigative reports available to the public / or reports have to be translated.

SOLUTION

 

Helm Order Monitor uses automatic speech recognition and, according to the maritime rules and regulations combined with data received from ship's sensors, monitors whether issued helm orders are clear, confirmed, and correctly executed.

The main benefits of this device are:

 

  • The helmsman will have to clearly repeat the helm orders and the device will issue visual and audio warnings if the helm order is being omitted, misinterpreted, or wrongly executed.

  • Assistance in monitoring the execution of helm orders when the concentration drops.

  • Indicator of the last issued helm order.

  • Implementation of automated warnings otherwise not available in hand steering, such as Default RoT deviation, Drift limit alarm, Max. rudder alarm and Low-speed alarm.

  • Transcription of recorder warnings data as an analytical tool for insurance companies from the maritime transportation sector, shipping companies, nautical schools/colleges, and training centers for seafarers.

HOM front - BLANK – NEW.jpg
HOM front - COMING SOON.jpg

WORKING IN THE DARK

Half of the time ships travel overnight, sometimes in low visibility conditions and relying on radar and lights from other ships. At night, any light pollution can make it difficult to spot approaching vessels. Screens must generate as little light as possible while still being easy to understand at a glance.

 

To minimize any excess light from the device, we designed a dark interface and paid special attention to color contrast to ensure legibility. The devices themselves were also chosen for their low wattage specs.

PRODUCT

 

Because a member of our team was a professional mariner for 30 years, observing the crew’s work and life on board brought him invaluable insights as to what is important as they worked hard to ensure safe passage to destination, and what challenges they grappled with.

Navigational technology in the maritime industry has not seen major upgrades in the last thirty years. The aesthetics of the tools onboard reflect that, widening the gap with modern smartphones that crew carry in their pockets.

 

Front of the device
image008 – NEW.jpg
Beamforming microphone
        (ceiling mounted)
Microphone – black.JPG
PARTS.JPG

Helm Order Monitor uses the small LED red dot matrix screen (for viewing distances up to 2 m) or 7” TFT touch screen (for viewing distances more than 2 m) to display information.

 

We designed the TFT version of Helm Order Monitor to facilitate ease of use and avoid “mistaps,” which is especially crucial in congested areas. We also worked to ensure gestures were familiar and intuitive, taking inspiration from the current equipment and technology the bridge team would be familiar with.

DuraMOn.bmp

AUDIBILITY

To develop a system that wouldn’t rely solely on sight, we turned to sound notifications. The noise level on the bridge is not to interfere with verbal communication, mask audible alarms or be uncomfortable to the bridge personnel. In this respect, the ambient noise level on the bridge in calm weather is not to exceed 65 dB(A).

Sound warnings consist of short buzzer signals that are sufficiently alarming and at the same time do not increase the cacophony of already existing warning sound signals on navigational bridges. The signals with a buzzer are as follows – examples are for helm orders by course:

  • One short signal - Drift limit alarm, Max. rudder alarm and Low-speed alarm.

 

  • One long signal - signals the helm applied on the wrong side or no action taken at all after issued helm order.

  • Two short signals – signals the following communication situations:

         - Helm order was misinterpreted by the helmsman.

         - The pilot or navigator did not acknowledge that the helm order was understood.

         - After executed helm order vessel is steady on the ordered course but the helmsman did not call out.

         - When the vessel is steady on the ordered course and the helmsman called out, the pilot or navigator do not acknowledge the helmsman's reply.

 

  • Three short signals – signals the following regarding ship's steering and course:

         - After issuing helm order, the helm moves to the correct side but the course changes slower or faster than the pre-set Rate of Turn speed (device settings).

         - After vessel is steady on the ordered course, she deviates more than 2 degrees.

         - The helm moves to the correct side but without helm order being repeated or acknowledgment that the helm order has been understood.

MENU

TAB 2.JPG

CONNECTION DIAGRAM

HOM connection - black.jpg

Mounting considerations: the main unit can be installed on a desktop, on the overhead, or flush-mounted in a panel. The beamforming microphone is ceiling mounted. The repeater can also be installed overboard with an optional waterproof box.

 

Generally, the installation and service are given to contractors with whom a particular company has signed shore-based maintenance of navigation equipment (SBM) contract.

MEASURES

Since there are 30 types of navigation equipment mandatory for installation on the navigational bridge, Helm Order Monitor is of minimal dimensions in order to fit more easily among the myriad of existing marine navigation equipment.

Desktop version measures
HOM s opcijskim nosačem #.jpg
HOM s opcijskim nosačem 2.jpg

WORKING EXAMPLES WHEN USING HELM ORDERS BY COURSE

Following helm order was issued: Port, steer one seven zero. The illustrations show the evolution of helm order by course.

 
HOM front - BLANK – NEW.jpg
image019.png
HOM front -  COURSE.jpg

HELM ORDER

Apart from extraordinary circumstances, the device activates only after repetition of the helm order (two-factor authentication) and subsequent acknowledgment by the pilot or navigator (Closed Loop Communication). This reduces the possibility of leading the helmsman.

REPETITION

HOM front - BLANK – NEW.jpg
HOM front -  WRONGLY REPEATED.jpg
image021.png

If the order is misinterpreted, the device displays WRONGLY REPEATED and emits a sound warning consisting of two short buzzes.

HOM front - BLANK – NEW.jpg
image023.png
HOM front -  LOOP NOT CLOSED.jpg

ACKNOWLEDGE

If helm order is not acknowledged by a pilot or navigator, the device displays LOOP NOT CLOSED and emits a sound warning consisting of two short buzzes.

HOM front - BLANK – NEW.jpg
image028.png
HOM front - WRONG RUDDER.jpg

WRONG HELM

After issuing the helm order the wrong helm was applied. The device displays a warning and emits a sound warning consisting of one long buzz.

 

Similarly, if the helm hasn't moved and the course hasn't changed device displays ORDER NOT EXECUTED and emits the same sound warning consisting of one long buzz.

STEADY COURSE

HOM front - BLANK – NEW.jpg
HOM front -  STEADY COURSE.jpg
image025.png

When the vessel is steady on the ordered course helmsman will call out. After the pilot or navigator acknowledge the helmsman's call, confirmation is displayed on the screen.

If the helmsman did not call out, the device displays COURSE NOT CALLED and emit a sound warning consisting of two short buzzes.

If the pilot or navigator does not acknowledge the helmsman's reply, the device displays COURSE NOT ACKNOWLEDGED and emit a sound warning consisting of two short buzzes.

image030.png
HOM front - BLANK – NEW.jpg
HOM front -  CHECK THE COURSE.jpg

DEVIATION

After the vessel is steady on the ordered course, she deviates more than 2 degrees. The device displays CHECK THE COURSE and emit a sound warning consisting of three short buzzes.

POLL - INCREASE THE RUDDER FEATURE

Although such an order does not exist in IMO Standard marine communication phrases, we would like to encourage ANYBODY involved in bridge operations to take this poll on the POLL page of this site.

Thru connected sensors it is possible to determine if the ship's Rate of Turn speed matches pre-set Rate of Turn.

This will trigger three short buzzes. We feel that addition of this feature would be beneficial for steering operations due to following reasons:

  • It will alarm if the rate of turn does not match pre-set ROT speed when issuing helm orders by course.

  • It will alarm if the rate of turn speed does not match ROT speed when issuing helm orders by Rate of Turn.

  • It will provide early warning if turn of the ship slows unexpectedly due to rudder position, wind, current, rudder malfunction, etc.

We will keep you updated on poll results through our newsletter.

PATENT

 

National Phase patent application. A patent application was filed in February 2021. to State Intellectual Property Office in Croatia.

 

GROWTH

  • Transcribed recordings of issued helm orders between helmsman and the pilot or navigator as additional evidence or a supplement to data from existing VDR or S-VDR devices - the practice has shown that in case of a maritime accident, audio recordings collected from Voyage Data Recorders are often not of sufficient quality to be relevant to the investigation, as stated in many publications.

  • Integration of captured audio recordings of helm orders into existing VDR or S-VDR devices.

  • Localization of automatic speech recognition in other languages used in maritime transportation.

  • Through the integration of water velocity databases, it is planned to offer early warning when closing to areas known for strong currents that could affect the ship's maneuverability.

  • Option for automatic plotting of CHL (Curved Heading Line) on Radar and ECDIS (Electronic Chart Display Information System) for improved radius control, as shown on the below image.

Curved heading line.JPG

TEAM

For the development of such an electronic device, it is necessary to join the maritime, information technology, electrotechnical engineering, and marketing sectors.

image037.jpg

Hrvoje Mihovilović (ELNAV) – founder/patent

Graduated in Maritime College in Split - Master unlimited. Professional mariner for 30 years in roles of Chief Mate, Navigation Officer, and Captain. Among others, he worked on passenger ships of Holland America Line, ro-ro passenger ships of Jadrolinija, and container ships of Mediterranean Shipping Company.

 

 

 

image038.jpg

Tomislav Seser (VOLTING) – hardware/software developer

An electrotechnical engineer with many years of experience in the design of high-level electrical systems. Also a programmer with advanced knowledge of the following technologies: PHP, SQL, HTML, CSS and JavaScript, MySQL databases, SEO (Search engine Optimization) and cloud technology.

 
image039.jpg

Nikola Tandara – (MELLOW) – software developer

Software engineer (FESB Split, Croatia), knowledge of JavaScript programming language, knowledge of work in c #. Net, CSS3, HTML5. Development and design of Frontend Interface for Earplay, admin panel Frontend, Earplay CANVAS control system, integration with Alex, rest API system for voice control.


 

image040.jpg

Ante Đonlić – computer programmer

Computer programmer with experience in programming languages: C #, Python, JavaScript, HTML, CSS and Java. Libraries and Frameworks: ASP.NET MVC, ASP.NET Core, entity Framework and entity Framework Core, Django Framework & Django REST, Node.js + Express.js.

image041.jpg

Mislav Biondić (MARKETING NUNDINAE) - marketing

Graduated in the School of Economics and Management in Zagreb. Director/owner of the marketing agency. The company's primary activity is market research, analysis, development of marketing strategy and promotion in corporate as well as the development of marketing strategy and promotion for smaller companies.