Full text of "Analysis of random errors in horizontal sextant angles."

See other formats

NAVAL POSTGRADUATE SCHOOL

Monterey, California

THESIS

ANALYSIS OF RANDOM ERRORS IN
HORIZONTAL SEXTANT ANGLES

Gerald B.

Mills

September

I98O

Thesis

Advisors:

. E
D.

N
W.

ortrup

Leath

Approved for public release; distribution unlimited

T197453

SeCUMlTY CLASSIFICATION OW THIS VaRC fW»>mm Dmtm Bitfrm^)

REPORT DOCUMENTATION PAGE

nt^oAT NUMScn

READ INSTRUCTIONS
BEFORE COMPLETING FORM

S. aOVT ACCIUIOM MO.

|. HCCIPiCmT'S CATALOG NUWaEff

« TITLE r«nOu*f((f«)

Analysis of Random Errors in
Horizontal Sextant Angles

I. Tv^e ow «e^oi»T * pcrioo covcmeo

Masters Thesis
September I98O

•. ^inroMMiNO ona. mi^out numsin

• . CONTHACT on CHANT NUMSCMC*)

7. AuTHonra;

Gerald B. Mills

to. PMOGHAM CLCMCNT. nnOjeCT, TASK
AntA * WOHK UNIT NUMCCns

I. Hen^OHMINO OnOANlZATIQN NAME AI^O AOQAII$

Naval Postgraduate School
Monterey, California 939^0

M CnNTWOLLINO Or^lCK NAME ANO ADOHCSS

Naval Postgraduate School
Monterey, California 939^0

12. nCPONT DATE

September I98O

II. NUMSCn OF ^ACES

14. MONITORING AGENCY NAME A AOOHESS/K rflffarwit trmm Cwiffolllfii Otlieu)

Naval Postgraduate School
Monterey, California 939^0

It. ttCUniTV CLASS, (ol iHI» riporl)

Unclassified

Mm. OKCUACllFlCATION/DOWNGnAOiNG
tCHEOULC

l«. OlSTAtauTlON STATEMENT (ol (Ma Ra»«rO

Approved for public release; distribution unlimited

17. OtSTNiauTlON STATEMENT fal ihm «Aarraef anlara^ in •l«e* 30, II dllttmtt hmm Km^tt)

It, SU»I>LEMENTAI«Y NOTES

It. KEY WOAOt CContlnua am r^rtmm at4» II nmemmamrr «n4 <tf«t«tliy trp Mack nuailMr)

Hydrographic Surveying; Sextant; Three-point Sextant Fix;
Horizontal Sextant Angles; Horizontal Positioning

20. ABSTMACT (Conllmf an

tmrr m*4 Id—tiltr tr tl««* nuaiAac)

The three-point sextant fix has been used for the horizontal
positioning of vessels in nearshore hydrographic surveys since
^775' However, this method has only recently been modeled
mathem.atically . The accuracy of the three-point fix depends on
the magnitude of the random and systematic errors in the angle
measurements and the fix geometry. The random errors in
horizontal sextant measurements were investigated by analyzing

DO I jAN*71 1*^73 EDITION OF 1 NOV •! IS OBtOLlTI

(Page 1) S/N ai03>OI«t AAOI

^^^iMMMMMtaib^^MtMi^Siai^kUUti^kM^j^t^m^austiL

over 1^00 angles, both at sea and on land. These random errors
were found to vary with the clarity of the signals being
observed, the stability of the vessel and the experience of the
observer. The upper and lower bounds for one standard deviation
were found to be about 2.7 and 0.6 minutes of arc respectively.
In addition, angular differences due to the direction of
rotation of the micrometer drum were examined as well as the
variability in the determination of sextant index error.

DD Form 1473 P

S/N 0102-014-6601 itcu»«T» clami^icatiom o^ tmu ^•ocrw**" o«»» f-«»'«*»

Approved for public release; distribution unlimited

Analysis of Random Errors in Horizontal
Sextant Angles

T^y

Gerald B. Mills
Lieutenant Commander, NOAA
B.A., Washington State University, I967

Submitted in partial fulfillment of
the requirements for the degree of

MASTER OF SCIENCE IN OCEANOGRAPHY (HYDROGRAPHY)

from the

NAVAL POSTGRADUATE SCHOOL
September I98O

c \

ABSTRACT

The three-point sextant fix has been used for the
horizontal positioning of vessels in nearshore hydrographic
surveys since 1775- However, this method has only recently
been modeled mathematically to quantify the effects of
various errors on fix accuracy. Positioning error in the
three-point fix depends on the magnitude of the random and
systematic errors in the angle measurements and the fix
geometry. Random errors in horizontal sextant measurements
were investigated by analyzing over 1^00 angular observa-
tions, both at sea and on land. These errors were found to
vary with the clarity of the signals being observed, the
stability of the vessel and the experience of the observer.
The upper and lower bounds for one standard deviation
were found to be about 2.7 and 0.6 minutes of arc respec-
tively. In addition, systematic errors resulting from
angular differences due to the direction of rotation of the
micrometer drum were examined as well as the variability in
the determination of sextant index error.

TABLE OF CONTENTS

I. INTRODUCTION 9

A. ■ HISTORICAL BACKGROUND 9

B. THREE-POINT FIX METHOD AND POSITIONING ACCURACY. 11

C. OBJECTIVES 1^

II. DATA COLLECTION METHODS 19

A. CRUISE DATA 19^

B. STATIONARY DATA 25

C. INDEX ERROR AND INSTRUMENT ERROR DATA 26

III. DATA REDUCTION PROCEDURES 30

IV. RESEARCH RESULTS 34

A. ABILITY OF THE OBSERVER 3/I

B. DISTINCTNESS OF SIGNALS . 36

C. PLATFORM STABILITY 38

D. DIFFERENCES DUE TO DIRECTION OF ROTATION

OF THE MICROMETER DRUM -^0

E. INDEX ERROR DETERMINATION l^-l

F. CRUISE II SYSTEMATIC ERROR t^,^

V. CONCLUSIONS k6

APPENDIX A. BLUNDERS AND SYSTEMATIC ERRORS AFFECTING

THREE-POINT FIX POSITIONING ACCURACY .... ^9

APPENDIX B. THEODOLITE INTERSECTION POSITION ERROR . . . 51^

APPENDIX C. DATA SET STATISTICS 60

BIBLIOGRAPHY 6?

INITIAL DISTRIBUTION LIST 69

LIST OF TABLES

TABLE I - Previous Values for the Magnitude of

Random Errors 15

TABLE II - List of Stations 21

TABLE III - Data Set Reference Code 33

TABLE IV - Mean Differences Between Simultaneously

Measured Angles 35

TABLE V - Statistics for Inverse Distances Between Main

Three-Point Fixes and Check Fixes 3^

TABLE VI - Comparison of Cruise Data to Wharf

Number 2 Data 38

TABLE VII - Index Corrections from Thirty Observations. • ^2

TABLE VIII - Instrument Errors • ^5

TABLE IX - Summary of Ranges of Sample Standard

Deviations ^6

TABLE B-1 - Theodolite Positioning Errors at Three

Locations '51

TABLE B-2 - Maximum Errors in Angular Best Estimates at

Three Locations Due to Theodolite Positioning
Errors 58

TABLE C-1 - Cruise I and Cruise II Data 61

TABLE C-2 - Wharf Number 2 Data - Without Scopes .... 62

TABLE C-3 - Wharf Number 2 Data - With Scopes 63

TABLE C-^ - Index Correction Differences ^^

TABLE C-5 - Abstract of Index Correctors from Cruise Data

and Wharf Number 2 Data 66

LIST OF FIGURES

Figure 1 . Three-Point Fix Geometry 12

Figure 2. Project Area and Station Locations 20

Figure 3* Template, Sextant and Tribrac -

Disassembled 28

Figure k. Template, Sextant and Tribrac -

Assembled 28

Figure 5- Data Distribution for ©i (Angle 110/205) -

Cruise I (top) vs. Cruise II (bottom) .... 37

Figure 6 . Distribution of Wharf Number 2 Data - Without

Scopes (top) vs. With Scopes (bottom) .... 39

Figure B-1 . Angular Error at Position 1 Due to 2 dj^ms

Theodolite Positioning Error of 0.59^ Meters . 56

ACKNOWLEDGEMENTS

I would like to express sincere gratitude to my
classmates for collecting the data which made this study
possible and to LCDR Donald E. Nortrup and LCDR Dudley W
Leath, my advisors, for their suggestions and assistance
I would especially like to thank my loving wife B.J. for
typing the numerous versions of this thesis and for her
understanding and patience throughout its preparation.

I. INTRODUCTION

A. HISTORICAL BACKGROUND

Instruments for measuring the altitude of the sun or
stars have existed for over 2,000 years, but it was not
until 1730 that the forerunner of the modern marine sextant
appeared. Two inventors, John Hadley of England and Thomas
Godfrey of Philadelphia, simultaneously designed instruments
using one fixed mirror and one movable mirror to measure
angles [May I963 ]. Both instruments allowed the movable
mirror to rotate through an arc of ^5° » but due to their
double mirror construction were able to measure angles up to
90°. Hence, they were called quadrants, although octants
would have been a more proper name. These quadrants were
not readily adopted by navigators and it was 1750 before
Hadley 's quadrant was in general use aboard vessels of the
East India Company [Cotter 1972] .

The need to measure angles greater than 90° prompted
Captain Campbell R.N. in 1757 to suggest the enlargement of
the arc of Hadley 's quadrant to 60° enabling the measurement
of angles up to 120°. Hence, the name sextant from the
Latin word sextans, "the sixth part" [Bowditch 1977] •

-t

Bowditch (1977) discusses the optics of this
construction which is similar to the modern sextant.

Although the sextant was designed primarily for
measuring vertical angles, it can be held on its side to
measure horizontal angles as well. Rev. John Mitchell first
suggested the use of the sextant for measuring horizontal
angles in hydrographic surveying in February ^7^5 [Cotter
1972]. Mitchell's method of fixing a vessel consisted of
intersecting the line of position derived from a horizontal
angle between two known points'^ and a position line obtained
from a compass bearing to one of the points. In l??^ » the
first Hydrographer to the Admiralty, Alexander Dalrymple ,
suggested determining position by intersecting the lines of
position derived from horizontal angles between three or
more known points [Cotter I972]. This principle had been
known by land surveyors since the early seventeenth century
but its application aboard ship was delayed due to the lack
of an accurate angle -measuring device .

The first application of Dalrymple 's suggestion was made
by Murdoch Mackenzie II. He surveyed the channels off the
Kent coast of England between l??^ and 1777 • Positions
could be plotted quickly by using a device called a three-
arm protractor or station pointer [ Admiralty Manual of
Hydrographic Surveying 1965]' This method for positioning

A known point is a reference station whose geodetic

coordinates have been determined. Also called a horizontal
control point.

is called the three-point fix or resection method and is
used today worldwide for inshore hydrographic surveys.

There have been several changes in the equipment used
in the three-point fix method over the last 200 years. An
endless tangent screw for continuous tracking and a micro-
meter drum for increased accuracy were added to the sextant
during the twentieth century. The marine sextant is fitted
with darkened shade glasses for observation of the sun. A
specialized sounding sextant has been developed specifically
for hydrographic surveying using lighter weight materials
and more rugged construction [Ingham 1975 ] • In addition,
the sounding sextant has a wide angle low magnification
telescope and a micrometer graduated in minutes of arc . In
recent years an electronic digital sounding sextant was
developed to enter observed angles directly into a mini-
computer aboard a vessel [Umbach 1976] . Despite these
improvements in the sextant the most draiiiatic equipment
change has been in the area of position plotting. Shipboard
computers and automated plotters have largely replaced the
three-arm protractor resulting in increased accuracy.

B. THREE-POINT FIX METHOD AND POSITIONING ACCURACY

Figure 1 illustrates the geometry of a three-point fix.
The known stations are depicted by triangles and labelled A,
B and C The vessel is located at point P and the observed
sextant angles are given by the symbols 0i and Oz- The

u
■p
o

B
o

■P

•H
O

0)
0
f>4

•H

Pl4

angle Qi and the points A and B define a locus of points
which is a circle through A, B and P. At any point on that
position circle, or line of position, the angle between A
and B is always e^. Similarly, there exists a position
circle for points B and C and angle 02 • A third line of
position through points A and C is defined utilizing the
angle ( 0i + 62) •

As can be seen in Figure 1, the three lines of position
intersect at point P. The accuracy of the position of P
depends on the errors in the measured quantities used for
position determination. Errors are classified into three
types; blunders, systematic errors and random errors.
Blunders are simply mistakes such as misreading a sextant
or improperly identifying a signal.^ They are eliminated
from the data by comparison with redundant or related
observations and careful editing. Systematic errors follow
some mathematical or physical laws and therefore have a
fixed relation to a set of conditions. For example,
elevation differences in observed signals cause systematic
errors in sextant measurements. In this case the conditions
(the elevation differences) can be measured and the correc-
tions can be calculated and applied. However, the laws
associated with systematic errors are not always known.

^ A signal is a natural or artificial object located at
a survey station (known point) which is used as a sighting
point for sextant measurements.

Systematic errors from unknown sources can be minimized by
following sound measuring techniques and instrument
calibration procedures. Random errors are inherent in all
physical measurements and can not be removed from the data.
Their effects must be estimated statistically. An example
of random errors would be measuring a known 50° angle
several times and obtaining sextant readings between
49°57.2' and 50^02. 8' .

Blunders and systematic errors affecting the accuracy
of positions determined by the three-point fix method are
discussed in Appendix A. The random errors in horizontal
sextant measurements at sea are the subject of this paper.

C . OBJECTIVES

Until recently, there have been very few attempts to
quantify the accuracy of sextant positions. Several
formulas have been developed in the last decade to do this
using the magnitude of the random errors as one of the
parameters. Tozzi (197^-) developed a series of equations
which included among other variables, the standard deviation
of the random error in measuring angles. Thomson (1977)
used the confidence intervals associated with the two angle
measurements of a three-point fix. The equation in
Bowditch (1977) includes the error in measurement of the
horizontal angles. Heinzen (1977) used the mathematical
notation for the standard deviation of the observed angles

in his development. Dedrick (I978) developed a formula for
the area of the error ellipse about some position given the
standard deviation of the sextant angle measurement error.

There seems to be no agreement among the various authors
as to the magnitude of these random errors. The table below
illustrates this.

Author

Year

Error

Goodwin

1973

6' 05.5"

Ingham

197^

+ 0.5'

Ingham

1975

+ 1 .0'

Thomson

1977

U.S. Coast

Guard

1978

several minutes

Dedrick

1978

few to several

minutes

Bodnar

1978

0.7' - 1.3'

TABLE I. Previous Values for the Magnitude of Random'
Errors

The errors mentioned by Goodwin, Thomson, Dedrick and
Bodnar specifically refer to the standard deviation of the
random errors while Ingham and the U.S. Coast Guard do not.
It should also be noted that Tozzi (197^) used Goodwin's
standard deviation in his development and Heinzen (1978)
used Ingham's 197^ value. All of the above errors are
estimates arrived at through experience except those of
Goodwin and Bodnar. Goodwin arrived at his result by
calculating the standard deviation of 32 angular measure-
ments between two well defined stations. He had 32
experienced navigating officers each measure the angle once

Bodnar's figures resulted from a one day experiment
aboard the NOAA Ship DAVIDSON while moored at a pier at
Lake Union in Seattle, Washington. Theodolite observations
were made from the ship to three easily identified objects
which were generally at the same elevation. Conditions of
extreme vessel stability due to tight mooring lines, little
wind and no tide enabled Bodnar to achieve agreement between
successive theodolite observations of about 0.5'- The two
angles (one of about 6^ and the other about 50°) were then
measured JO times each by six officers. The means and
standard deviations were calculated for each of the 12 data
sets. All of the means were within 0.5' of the angles
determined by the theodolite measurements. The individual
standard deviations ranged from 0.?' to 1.5'' No cumula-
tive statistics -were determined.

Dedrick (1978) attempted to use historical data to
arrive at his estimate. During a survey of south San
Francisco Bay between 1857 and 1858 it was common practice
to take full rounds of angles at a station. That is, angles
were measured between objects all around the horizon. He
studied 17 rounds of angles each consisting of four to
seven sextant angles. The disagreement between each round
and 3^0° was generally between 10' and 20' of arc, but
ranged from 2* to 55' of arc. He states that "this data
would suggest that as an extreme upper limit, values of
of a few to several minutes of arc might be appropriate."

The main objective of this study was to quantify the
random errors in sextant angle measurements under varying
conditions. The factors upon which these errors are depen-
dent are: (1) the ability of the sextant observer,

(2) the visibility and distinctness of the signals, and

(3) the stability of the platform. The ability of the
sextant observer was analyzed by comparing more experienced
observers with those with less experience. All observations
were made across water on clear, somewhat windy days so that
horizontal refraction was at a minimum. The distinctness of
the signals was altered by using telescopes on the sextants
for some observations and not for others . Platform sta-
bility was analyzed by measuring angles under three
conditions: (1) vessel in moderate to heavy seas,

(2) vessel in calm seas, and (3) observer at a stationary
point on a wharf.

During the course of data collection a few other
questions arose which were related to random errors. Is
there a difference in angle measurement if the micrometer is
turned clockwise or counterclockwise? How much do individ-
uals vary when determining index error on the same sextant?
Are the manufacturer's stated instrument errors correct?
Answers to these questions and the analysis of the random
errors are discussed in later sections.

The scope of this paper does not include the positional
accuracy of three-point sextant fixes. Tozzi (197^) »

Heinzen (1978) and especially Dedrick (1978) covered this
subject in detail. No attempt was made to evaluate the
random errors while a vessel was underway as it would be
when running a hydrographic survey line. The data analysis
in this thesis corresponds to the use of the three-point fix
for calibrating or evaluating electronic positioning control

II. DATA COLLECTION METHODS

Raw data for this study consists of sextant angle
measurements observed in the southern portion of Monterey
Bay, California. This area and the horizontal control
stations used during the project are shown in Figure 2.
All of the station positions were determined by third-order
methods or better by personnel of the National Ocean Survey.
These stations, their positions, elevations and station
numbers and the sources of this information are shown in
Table II. The data was collected both at sea and on land
and will therefore be discussed separately. All sextants
were checked each day for adjustable errors and were found
to be satisfactory.

A. CRUISE DATA

The 126-foot long research vessel ACANIA was used as
the observation platform for the anglemen (sextant
observers) during the two data collection cruises. The
position of the ship was determined for every sextant
observation by the standard theodolite intersection method
described by Umbach (1976). Briefly, this method consists
of occupying two horizontal control stations with
theodolites. Each theodolite operator measures the angle
between another known point and the object to be located.

121* 54- W

I
121* 53-W

AMERICAN CAN

CO. STACK A

(202) ^-^

KMBY MAST
(203)

MONTEREY

2^26-U

MONTEREY CO. DISK (301)

'MONTEREY HARBOR LT. 6 (205)

FT. ORD SILVER

WATER TANK (206)

use: mon ^

■ (110)

NPGS TOWER A
(302)

WW'^M CRUISE I
^^^ CRUISE n

Figure 2. Project Area ajid Station Locations

»-H

T-1

Cv.'

fil

C/0

«-<

o w

> w

cvj

r^ vPi \o

v>,

«-l

c--

^J^

CJN

•

t-l

>J^

:r)

«-l

♦H

vr>

1-t

>J-^

<-i

1-1

«-i

r-t

♦H

t-(

«-l

h-J

T-l

C3N

CN-

•

O-N

1-1

u-\

E-i

o->

E-*
C/3
O

ir>

Eh
k5

•

C3N

•

!=)

t/J

CJN

jrl

•

t/3

Cx.

t:d

o cc

M W

»-l

EH PQ

U>1

«-l

< s

»H

EH&

W 2

0
w

•H

13
0)

-P
CS

•H

W
W

«H
NO

E
o

c
a>
cd
«

C
o

•H

a
+j
w

•H
CO
•H

E-»

Since the azimuth can be computed between any two known
points, the azimuth to the unknown point can be determined.
Therefore, the theodolite measurements from each station
produce lines of position, the intersection of which is the
location of the unknown point. To ensure the correctness
of the beginning azimuth, a check azimuth to a second known
point is usually observed.

Stations MUSSEL (10?) and MONTEREY CO. DISK (301) were
occupied with Wild T-2 theodolites on both cruises. USE
MON (110) was observed to obtain the initial for MUSSEL
(107) and MONTEREY HARBOR LT 6 (205) was the azimuth check.
The theodolite observer at MONTEREY CO. DISK (301)
initialled on IVIUSSEL (10?) and sighted on USE MON (HO) as
the azimuth check. (Subsequent references to these and
other stations will only be by the station numbers indicated
in Table II.) Both theodolite operators attempted to use the
center of the three anglemen as a target. However, since
this was not easily visible at all times, a bright orange
float, 2 feet in diameter, was tied to the ship's rail
beside the observers.

The sextant observers consisted of six individuals with
three levels of experience in measuring angles with sex-
tants. Two anglemen (1 and 2) had extensive horizontal
angle measurement experience. Two others (3 and ^) had
extensive experience with measuring vertical angles with a
sextant, but none with measuring horizontal angles. The

final two (5 and 6) had no prior sextant experience. The
six observers were divided into two groups of three. The
more experienced group consisted of the first two observers
and anglemen number 3- Each group of three measured angles
simultaneously from the aft portion of the upper deck of
ACANIA. The observers stayed within one meter of each other
to minimize the effects of eccentricity.

Observations from the ship were coordinated with those
ashore by using three portable CB radios. To reduce the
positioning errors caused by theodolite mispointings the
captain of ACANIA attempted to stay within a small area
about ^50 meters NNW of the Coast Guard pier. This position
minimized the distance to the two theodolite locations and
gave an intersection angle of near 90°. Unfortunately,
weather conditions made it impossible to maintain station.
Figure 2 shows the areas of operations for both cruises.

Cruise I took place under adverse weather conditions.
Winds were 20-25 knots from the WNW with seas of
approximately 5 to 8 feet. ACANIA, with its 126-foot length
and 22-foot beam, provided a very unstable platform. By
not using scopes on the sextants under these conditions of
poor platform stability, the largest value for the standard
deviation of the random errors was obtained.

Each angleman measured the angle between signals 110 and
205 a total of 32 times. In addition, the more experienced
group measured the angles required to compute ten pairs of

three-point fixes. Each pair of fixes was derived from a
left angle 0i(llO/2O5), a right angle 02(205/202) and a
right check angle 0^ (205/10?). The fix determined from
angles Qi and ©2 was designated the main fix and the fix
computed from 0^ and Q-^ was called the check fix. The
distance from the main fix to its corresponding check fix
was determined and is called the inverse distance. Blunders
in the horizontal angles can "be detected by analyzing the
size of these inverse distances. If the values are less
than five meters, then the National Ocean Survey considers
that no "blunders have been made. For the ten pairs of fixes
from Cruise I the mean inverse distance was 1.218 meters
with values ranging from 0.003 to 2.390 meters (standard
deviation = 0.80 meters). Planned operations to collect
additional angle data for three-point fixes were cancelled
due to worsening weather conditions and failing radios. A
total of 222 sextant angles were collected for analysis.

The weather conditions for Cruise II were much better
than those for Cruise I. The wind was from the southwest
at 5 to 10 knots with no appreciable seas. However, long
period swells of 1 to 2 feet caused some vessel motion.
Scopes were used on the sextants in an attempt to determine
a reasonable least value for the standard deviation of the
random errors at sea. It was planned to also collect data
with no scopes, but a heavy rainstorm reduced visibility to
less than 1,000 meters.

On this cruise each angleman took 32 measurements of
the angle 110/205 • As in Cruise I, angles were observed to
compute pairs of three-point fixes. The same signals were
used for both cruises. Each group measured a total of 96
angles resulting in 32 pairs of fixes. The mean inverse
distance was 0.7^9 meters with values ranging from O.Oll
to 2.555 meters (standard deviation = 0.61 meters). The
total number of angles measured for Cruise II was 38'^*

B. STATIONARY DATA

All of the data collected at sea was subject to possible
errors from the observers not being in the exact same
location and from the theodolite intersection method. These
errors are discussed in Appendices A and B respectively.
The random errors were also influenced by platform
stability. To evaluate the effects of this factor the
observers measured a series of angles from station 350 on
Wharf Number 2 at the Monterey Harbor. A T-2 theodolite was
used at 350 to measure the horizontal angles between 110 and
206 and between HO and 302. These angles were measured
according to the third-order specifications prescribed by
Umbach (1976) which require four measurements with different
plate settings, all within 5 seconds of the mean.

Each of the 6 anglemen stood directly above 350 and
measured both angles 30 times with scopes and 30 times
without scopes. In addition, each 30 observation set was

divided such that 15 angles were measured with the
micrometer drum being turned clockwise or decreasing in
value and 15 angles were measured with counterclockwise
motion. Since the observed stations were not at the same
elevation as 350 i the observed angles were not horizontal
angles. The equation from Umbach (1976) for converting
inclined angles to horizontal angles and vice versa is
given in Appendix A. The angular elevations of each object
at 350 were measured with a T-2 theodolite. Hence, the
horizontal angles 206/110 and 110/302 were converted to the
inclined angles that were observed by the sextants. The
errors were then calculated for each of the 720 sextant
angles.

C. INDEX ERROR AND INSTRUMENT ERROR DATA

Tests were conducted to determine if any systematic
errors were unaccounted for in the previous data. Each
individual determined index error for his sextant every
time he made a set of observations. However, there were
larger than expected differences of index error between
individuals using the same sextant. Individual determina-
tions also varied from day to day. Therefore, index errors
were studied by having each angleman make 30 measurements
of index error with each of the three sextants. The
procedure consisted of holding the sextant vertically,
observing the sea horizon and bringing the direct and

reflected images into coincidence. As before, 15
measurements were made with a clockwise micrometer drum
movement and 15 with a counterclockwise movement. A total
of 5^0 index error measurements were analyzed.

Instrument error was also analyzed for each sextant.
Recall that instrument error consists of graduation error
and centering error. This is determined for each instru-
ment by the manufacturer and the results are posted inside
the sextant case. Each of the three sextants used on this
project had posted correctors of 0.0 minutes for each 15°
increment of arc. To check some of these values a method
suggested by CDR . James Wintermyre of the NCAA Pacific
Marine Center's Operation Division in Seattle, Washington,
was used. He reasoned that to properly "calibrate" a
sextant or determine its instrument error for various
angles, accurate theodolite angles must first be observed
between objects. The sextant could then be used to measure
the same angles. To maximize the accuracy of this -calibra-
tion the sextant must be perfectly horizontal and over the
identical point from which the theodolite angles had been
measured. To accomplish this he designed a sextant template
that would mount in a T-2 theodolite tri-brac and would
accommodate three different makes of sextants. The template,
tri-brac and a sextant are shown in Figures 3 and ^.

Initially a temporary reference station was established
from which several objects at the same elevation could be

Figure 3- Template, Sextant and Tribrac - Disassembled

Figure k. Template, Sextant and Tribrac - Assembled

sighted on. The objects were the southwest edge of a
chimney, a flagpole on Wharf Number 2 and the southern edge
of the AMERICAN CAN CO. STACK (202). Their elevations were
determined to be the same as the theodolite by setting 90°
on the vertical circle. The angles between the objects
were measured with the theodolite, again to third-order
specifications. Unfortunately, the theodolite did not
have a removable tri-brac . Hence the template was mounted
on another tri-brac which was centered over the station.
The "bull's-eye" level bubble of this new tri-brac was out
of adjustment. But since the proper vertical location on
each target was well determined by the theodolite, the
tri-brac was adjusted to allow sighting on these points
with the sextant. Each angle was measured 30 times with
each sextant - 15 times with a clockwise micrometer movement
and 15 times in a counterclockwise direction. Thus, a total
of 180 angles were observed for analysis.

III. DATA REDUCTION PROCEDURES

All computer work for this project was done on the
IBM 360/67 system at the Naval Postgraduate School's W.R.
Church Computer Center. All programs were written in
FORTRAN IV. Program UCOMPS was used to determine positions
from three-point sextant fixes and T-2 theodolite inter-
sections. It was provided by the National Ocean Survey -
NOAA, Rockville, Maryland. Program INVERS was utilized to
compute the lengths and azimuths of lines between known
points. This program was obtained from the National
Geodetic Survey - NOAA, Rockville, Maryland. Library
routine HISTF was used for the statistical analysis of all
data for this project.

Error analysis requires that the best estimate of a
measured quantity be determined. Usually this value is the
mean of a number of measurements of a particular angle or
distance. However, the sextant angles observed during the
two cruises could not be treated in this manner since ACANIA
was always moving. Thus each angular measurement was a
unique and unrepeatable observation. Therefore, the best
estimate of each sextant angle observed aboard ACANIA was
derived utilizing the theodolite determined ship positions.
Inverses were computed from each position to the horizontal
control stations upon which the sextant observations were

made (station numbers 110, 205, 202 and 107). The
difference in forward azimuth to any two stations from the
ship's position was chosen the best estimate of that angle.
Likewise, the best estimates for the sextant angles observed
at Wharf Number 2 were the inclined angles derived from the
T-2 theodolite measurements there (see page 26).

Errors ( e ) were calculated for each observed sextant
angle (x) by subtracting the best estimate of the angle (X) .
This is expressed mathematically as e j_ = Xj_ - Xj^ where
i is the number of the observation. The mean ( i ) of the

errors for each data set was then calculated as follows:

n
e= .Z.z-/n where n is the total number of observations. If

1=1 T

the errors in the sextant angles are normally distributed
and totally random, then by definition e must be equal or
nearly equal to zero. If e does not equal zero for each
data set, then sampling errors and/or undetermined
systematic errors were present. The bounds of sampling
errors for a given probability are directly proportional to
the sample standard deviation of the data set and inversely
proportional to the square root of the number of errors
analyzed.

The sampling and systematic errors must be eliminated
when calculating the sample standard deviation ( a ) of the

The errors determined for each observer under each
set of conditions are referred to as a data set.

random errors. This was done for each data set by using

I n 9
the equation a = .[ Z (e. -e) ]/(n-l)

\| i = l

It can be shown that a is a measure of precision and e is
a measure of accuracy .

The mean of the errors ( i ) and the sample deviation
( a ) for each data set and for some selected combination
of data sets were calculated. These results are discussed
in the next chapter and are summarized in Appendix C .
Graphs of the distribution of the errors for some of the
combined data sets are also shown. These graphs called
frequency polygons are formed by connecting the mid-points
of the tops of the bars in the histograms of the data.

A coding system was devised to simplify references to
various data sets or data set combinations. The code
consists of four or five characters and is shown in
Table III.

The fifth character of the code is used for the cruise
data if angle Qi was reobserved when collecting data for
three-point fixes. Hence, data set INl - 0;[(2) would refer
to the second data set collected by observer 1 while mea--
suring angle Q^ with no scope on Cruise I. Combinations
of data sets are referred to by the part of the data that
is common to all sets. For example, combination data set
IN - Qi would refer to all no scope observations of angle
01 on Cruise I.

(1) Data Origin:

I - Cruise I

II - Cruise II

W - Wharf Number 2 (Stationary)
B - Beach Lab (Index Error Data)

(2) Use of Scopes on Sextants:
N - no scopes

S - with scopes

(3) Observer Number (experience level decreases as

this number increases:

1 through 6
(^) Angle Designator:
% - 110/205
®2 - 205/202
03 - 205/107
% - 206/110

e^ - 110/302

(5) Repeated Data Set (if necessary):

(1) first set

(2) second set

TABLE III. Data Set Reference Code

IV. RESEARCH RESULTS

The factors affecting the random errors in horizontal
sextant measurements are the ability of the sextant
observer, visibility and distinctness of the signals and
the stability of the platform. The effects of these factors
are not independent, and it is therefore impossible to
isolate the contribution of each error source. In addition,
errors induced by the theodolite intersection method and
errors due to eccentricity of the observers are included in
all of the data collected at sea. Statistics for all data
sets are summarized in Appendix C

A. ABILITY OF THE OBSERVER

This factor is best evaluated by comparing the results
of the data sets for each observer under similar conditions.
The mean of the errors and the sample standard deviation are
denoted by i and a respectively. Cruise I data is
denoted by data sets beginning with IN in Table C-1 . The i
for observer six seems extremely large. This was likely
due to the seasickness that the individual experienced while
measuring angles. As mentioned before, the sea conditions
for Cruise I were extremely rough.

The data from Cruise II is shown in Table C-1 beginning
with code IIS. The strong positive bias of the e's will be
discussed in section F.

The results of Zk data sets taken at Wharf Number 2
are indicated in Tables C-2 and C-3 by the codes beginning
with WN and WS . These data sets show no major differences
between the experienced and inexperienced observers.

Observer experience was also evaluated by comparing
simultaneously observed sextant angles. This eliminated
the effects of positioning error from the theodolite inter-
sections. The mean of the differences for each group is
shown below in Table IV for both cruises. Group 1 consisted
of observers 1-3 and group 2 of observers k-6 . Two angle
differences from group 1, Cruise I were rejected due to an
obvious blunder in one angle of 10'. All angle differences
involving observer 6, Cruise I were rejected due to their
unreliability. The values in parentheses indicate the
number of differences used to compute each mean. Some
error was introduced due to the eccentricity of the three
observers in each group. However, this error was assumed
to be the same for each group.

Cruise I Cruise II

Group 1 2.5^4- (19) 1.29 (96)
Group 2 3.10 (32) 1.66 (96)

TABLE IV. Mean Differences Between Simultaneously
Measured Angles (Minutes of Arc).

The agreement of the group 1 observers was 0.3' "to 0.5'
better than that of the group 2 observers. This was the
best indication of improvement in measuring horizontal
sextant angles due to increased experience.

B. DISTINCTNESS OF SIGNALS

The effect of signal clarity or distinctness was
determined by comparing data collected without scopes
(Cruise I) on the sextants to that collected with scopes
(Cruise II). The data for G^ (angle 110/205) from Cruises
I and II are denoted by IN- 0^ and IIS- 0^. The cumula-
tive data from these data sets is shown graphically in
Figure 5- This illustrates the increased dispersion of
the Cruise I data compared to that of Cruise II.

•As a further comparison, inverse distances were
determined between the main three-point sextant fixes and
the corresponding check fixes. The results are shown
below in Table V. Again the number of observations used in
computing the statistics are shown in parentheses.

Cruise

I (10)

II (6^)

1.22

0.75

0.80

0.61

TABLE V. Statistics for Inverse Distances Between Main
Three-Point Fixes and Check Fixes (Minutes
of Arc) .

0.4

0.3

>-
o

§0.2

LU
0£

iN-e,

NP=167
i=-0.32
a= 2.69

0.1

0,0 III' ymtr-^^/x

-10.0

0.4 r.

0.3

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

go. 2

0.1

0.0

IIS-9,
NP=256
£=+1.63
a= 1.58

-10.0

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

Figure 5- Data Distribution for 0i (Angle 110/205) -
Cruise I (top) vs. Cruise II (bottom)

The decreased dispersion in the data from Cruise II
versus that from Cruise I is due not only to the use of the
scopes on the sextants, but also to the decrease in vessel
motion. The data at Wharf Number 2 eliminated vessel
motion completely. There were dramatic decreases in the
values of e and a when scopes were added to the sextants.
The cumulative statistics of the data are illustrated in
Figure 6. Hence, the use of scopes on sextants decreases
the magnitude of random errors in horizontal sextant angle
measurement.

C. PLATFORM STABILITY

The effects of platform stability on random errors are
best evaluated by comparing the Cruise I data to the data
collected on Wharf Number 2 without scopes and comparing
the Cruise II data to the data collected with scopes on
Wharf Number 2 . These four sets of data are shown in
Figures 5 and 6. The statistics are summarized below in
Table VI.

Cruise I

Wharf 2
(No Scopes)

Cruise II

Wharf 2
(Scopes)

-0.32

+0.89

+ 1.63

+ 0.20

2.69

2.07

1.58

0.6^

TABLE VI. Comparison of Cruise Data to Wharf Number 2
Data (Minutes of Arc).

0.4

0.3

NP=360
£=+0.89
a= 2.07

0.4 r

0.3

0.2

-10.0

-5.0 0.0

ERROR (MINUTES)

10.0

0.1

0.0 I ' t ' I I I I I I I I I

NP=360
G=+0.20
a= 0.64

rJ-f>^ i I I i I I l_J I I L_J L.

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

Figure 6 . Distribution of Wharf Number 2- Data - Without
Scopes (top) vs. With Scopes (bottom)

The increased dispersion of the cruise data is apparent
in the figures as well as the table. Hence, when vessel
stability is increased there is a significant reduction in
the magnitude of random errors.

D. DIFFERENCES DUE TO DIRECTION OF ROTATION OF THE
MICROMETER DRUM

The differences in the Wharf Number 2 data created by
clockwise rotation of the micrometer drum versus counter-
clockwise motion are shown in Tables C-2 and 0-3-
Observations 1-15 were made with clockwise micrometer drum
movement and correspond to decreasing values. Counter-
clockwise rotation was used for observations I6-30. The
magnitude of the differences in sample standard deviations
is small - none are greater than 0.20' for the data taken
with scopes and only 3 sore greater than 0.^0' for the no
scope data. Observers 1 , 3 and 5 seem to measure smaller
angles when using counterclockwise micrometer rotation.
These three all have negative i values with average Ae's
of -0.68', -0.60' and -0.50', respectively. The other
observers did not exhibit this tendency as strongly.

The procedure for collecting index error information
at the Beach Lab property was described on page 27 . The
corresponding index correction (the negative of index
errors) are summarized in Table C-A-. Scopes were used on
the sextants. As before, observations I-I5 were made with
clockwise micrometer drum movement and observations I6-30

^-0

with counterclockwise rotation. The magnitude of the
differences in sample standard deviations is again small -
all are less than O.I3' • Observers 1, 3 and 5 again measure
slightly smaller angles when using counterclockwise micro-
meter rotation. Their average Ai'sare -0.20', -O.32' and
-0.48', respectively. In addition, observers 2 and k have
negative Ae's. Therefore, the direction of rotation of the
micrometer drum may introduce some small systematic error
into sextant angle measurements.

E. INDEX ERROR DETERMINATION

Index errors were determined by each observer every day
before measuring angles. The index corrections for this
project are summarized in Table 0-5- Each index correction
was derived using the method described by Umbach (1972).
This procedure consists of holding the sextant vertically,
observing the sea horizon, bringing the direct and reflected
images into coincidence and reading the micrometer and
vernier. This is repeated several times, alternately
turning the micrometer drum clockwise and counterclockwise.
The results are averaged, and this average is the index
correction. From Table C-5 it is seen that index correc-
tions for the same sextant vary between individuals by as
much as two minutes. Even the same observer had differences
of up to 1.7 minutes from day to day.

Instead of this method of daily determination of index
corrections, the observations in Table C-4 can be averaged
for each observer. The results of this are shown below
in Table VII.

Observer 12 3^56

Sextant #2972 -O.50 -0.68 -0.86 -0.^6 -O.50 -0.74

Sextant #2982 -O.^il- -0.26 -O.^-k -0.25 -O.32 -0.53

Sextant #3003 -O.39 -O.38 -0.64 -O.09 -O.50 -0.64

TABLE VII. Index Corrections from Thirty Observations
(Minutes of Arc) .

These results show differences from the index
corrections determined by the daily method of as much as
1.5' indicating a systematic error. If there were no
changes in index correction during the project the averages
in Table VII should be more accurate than the daily
determined correctors. Applying these newly determined
index correctors would alter the sample means of all the
previous data, but not the sample standard deviations.
These observations suggest that each angleman should
determine an index correction for his sextant by averaging
ten to fifteen measurements in each direction thereby
minimizing the magnitude of the systematic error.

F. CRUISE II SYSTEMATIC ERROR

Table C-1 showed that the observations of angle 110/205
during Cruise II had a positive bias of about 1.6'. Several
possible error sources for this systematic error were
considered and are described below:

(1) Mislocation of Signals 110 or 205: This did not
occur since the angles between these stations during Cruise
I showed no such positive bias.

(2) Consistent Errors in Theodolite Positioning: This
could only occur if the theodolites were mispointed on the
initial. This is highly unlikely due to the large number
of such pointings and the good agreement with the azimuth
check stations.

(3) Signals at Different Elevations: The excess angle
measured due to differing elevations ranged from 2.5" to

3 '5" throughout the Cruise II work area. This was much
smaller than the 1.6' bias.

(^) Incorrect Index Corrections: It is very unlikely
that all six sextant observers could have made the same
large mistake .

(5) Collimation Error: This error occurs when the
scope is not parallel to the frame of the sextant and is
always positive. No detectable collimation error was found
when the sextants were examined using the method outlined
in The Admiralty Manual of Hydrographic Surveying (I962).

(6) Incorrect Instrument Error: This value is
determined for 15 degree increments of every sextant by
the manufacturer and is attached to the inside of each
sextant case. All three sextant cases had identical instru-
ment errors posted for each angle - 0.0*. This suggested
that the sextants should be checked for instrument errors.

The method suggested by CDR- Wintermyre and described
earlier on page 27 was used in an attempt to evaluate the
instrument error of all three sextants . Thirty observations
were made with each sextant on two different angles. As
determined by theodolite, angle one was 28051'22' and angle
two was 47°03.00'. The differences between the theodolite
measured angle and the sextant angle should only depend on
theodolite error, index error and instrument error if the
sextants are in otherwise good adjustment. The set of four
theodolite measurements for each angle resulted in spreads
of 03.6" and 07-8" respectively. Hence, theodolite errors
were minimal. The mean index corrections that were deter-
mined from 180 measurements for each sextant were -0.^0' for
sextant #2972, -O.35' for #2982 and -0.45' for #3003. The
index corrections arrived at by the author before using the
sextant template were -0.66', -0.3^' and -0.5^' for the
same three sextants. The difference between the theodolite
angles and the mean of the JO sextant angles (corrected for
the author's index corrections) are shown in Table VIII.

Sextant Difference (Angle One) Difference (Angle Two)
2972 -0.72' -0.55'

2982 -0.65' -O.9O'

3003 -0.9I' -0.55'

TABLE VIII. Instrument Errors

It should be emphasized that the values in Table VIII are
errors, not corrections. Therefore, the angles measured by
the three sextants are 0.5' to 1.0' too small. This is
contrary to the results from Cruise II which showed the
sextant angles to be about 1.6' too large.

In summary, all the possible errors have been
considered. The 1.6' bias results from some unknown source.
Nevertheless, the standard deviation of the random errors
from Cruise II is not affected by this error in the sample
means.

V. CONCLUSIONS

The standard deviation of the random errors in
horizontal sextant measurements was found to vary mainly
with the clarity of the signals being observed and the
stability of the vessel. The differences due to observer
experience were quite small for both the cruise data and
the Wharf Number 2 data. Signal clarity was evaluated by
measuring angles both with and without scopes on the sex-
tants. This was isolated for only the Wharf Number 2 data.
The cruise data showed the combined effects of signal
clarity and platform stability and included some unknown
amount of error due to theodolite positioning.

The general range of the sample standard deviations
from the experimental data are summarized below in Table IX

Platform Stability Scopes/No Scopes Sample Standard

Deviation

Unstable (Cruise I) No Scopes 2. 3 '-3. 6'

Somewhat stable Scopes 1.3'-1.8'
(Cruise II)

Very Stable No Scopes 0.9' -2.0'
(Wharf Number 2)

Very Stable • Scopes 0.3' -0.8'
(Wharf Number 2)

TABLE IX. Summary of Ranges of Sample Standard
Deviations.

Although the above table illustrates the range of
values for all observers, the cumulative statistics for all
the data collected under each set of conditions give a
better estimate of the magnitude of the random errors. The
sample standard deviation for all angles measured with no
scopes on an unstable platform is 2.69' and represents the
probable upper bound. The lower limit for the sample —
standard deviation was derived from the Wharf Number 2 data
with scopes and is determined to be 0.6^'.

Index corrections for a given sextant may vary from
individual to individual, but probably not by more than
0.5' • Some individuals measure consistently smaller or
larger angles (by as much as 0.6') depending on the direc-
tion of rotation of the micrometer drum. Therefore, for
increased accuracy, index corrections should be determined
by averaging ten or more measurements in each direction.

Some suggestions for further work in this area are
appropriate. By using three T-2 theodolites to obtain a
least square adjusted ship position, a more accurate set
of statistics could be obtained for angles measured at sea.
Various types of signals could be used at station locations
to further determine the dependency of random error on
signal clarity. The most variable quantity in attempting to
quantify the random errors in horizontal sextant measure-
ments is the ability of the observer. Although over 1^00
horizontal angles were analyzed, only six sextant observers

were used. Further work with different observers would
give a broader data base. Nevertheless, this study does
present analytically derived values for the standard devia-
tion of random errors in horizontal sextant measurements
where only estimates existed before.

1^8

APPENDIX A: BLUNDERS AND SYSTEMATIC ERRORS
AFFECTING THREE-POINT FIX POSITIONING ACCURACY

The potential blunders associated with three-point fixes
include the following:

(1) Misread Sextant: This blunder is not readily
identified for individual fixes. However, when conducting
a hydrographic survey, consecutive fixes fall in a straight
line if the vessel is carefully steered. A misread sextant
angle will cause the fix to deviate from this line . The
fix is then either rejected or an artificial position is
created using dead reckoning.

(2) Misplotted Fix: This generally occurs only with
manually plotted fixes and is identified by the same method
as the error above. It is corrected by simply replotting
the fix.

(3) Improper Identification of a Signal: This error,
like the above two errors, is not easily discovered when
only one fix is taken. Even along a carefully steered
survey line, it may go undetected if the same erroneous
signal is used throughout. But if the observer switches
from that signal to a correct signal while on line the
resulting fix will deviate from the straight line created
by the previous fixes. The sextant data may be retained
and the correct positions determined if the misidentif ied
signal can be properly identified.

The systematic errors that result in reduced accuracy
of three-point fixes are as follows:

(1) Weak Fix Geometry: Strong geometry exists for a
three-point fix when two of the three lines of position
intersect at right angles. A fix has weak geometry when
the three lines of positions approach coincidence. This
greatly increases the effect of other errors on positional
accuracy. Various fix geometries are discussed by Umbach
(I976), Bowditch (1977) and Dedrick (1978). The effects

of weak fix geometry are minimized by following the general
rule specified by these authors.

(2) Station Positions in Error: This error is similar
to misidentifying a signal and its detection is also similar
if the station position error is large. Small errors in
station positions will often be undetected and will always
be present in any three-point fix. Heinzen (1977) and
Dedrick (1978) both discuss the three-point fix positional
errors caused by incorrect station positions.

(3) Phase Error: The apparent displacement of a
signal due to unequal illumination of its surface is called
phase error. It is dependent on the shape of the signal,
the angle of the sun with the line of sight, and the
intensity of the sunlight. Water tanks may be especially
susceptible to this kind of error. Formulas for correction
of phase are usually not practicable due to the numerous
factors upon which the correction depends [C-ossett 1971] •

(^) ' Observer and Observed Signals Not at the Same
Elevation: The angle observed between signals with
elevations differing from that of the observer are called
inclined angles. This error is minimized by choosing
signals that are at the same elevation as the observer. If
this is not possible the inclined angle can be reduced to a
horizontal angle by using the following formula from
Umbach (1976):

cos C = CQs Q - sin(hi )sin(h?)
cos (hi) cos (h2)

C = the horizontal angle

0 = the observed inclined angle

hi = the angular elevation of station 1 above the

observer
h2 = the angular elevation of station 2 above the

observer

(5) Two-Observer Eccentric Error: This error is caused
by the angle observers not being at exactly the same point.
The magnitude is dependent on the distance between the
observers and the angle of intersection of the two lines of
position. Dedrick (I978) discusses this error and shows
that for a separation distance of 3-0 feet and a 50° angle
of intersection, the maximum error in the position is about
^.3 feet. It is minimized by selecting strong fix geometry
and by having the angle observers stand as close together

as possible.

(6) Horizontal Refraction: Differences in the density
of air along a line of sight can cause bending or refraction

of light rays. Vertical refraction is usually larger than
horizontal refraction due to the air being stratified with
denser layers near the ground. These layers are not hori-
zontal over terrain that is sloping or unevenly heated and
hence, horizontal refraction occurs. A line of sight pass-
ing partly over water and partly over land is an example of
unevenly heated terrain. Errors due to horizontal refrac-
tion can be as large as 10 to 18 seconds of arc [Gossett

1971] .

(7) Sextant Parallax: This is caused by the separation
between the center of the index mirror and the line joining
the telescope axis and horizon glass (usually about ^ to 6
cm). It decreases as the range to the station increases.
For a separation of ^.3 cm the parallax correction decreases
from 0.^9' of arc at 1000 feet to 0.05' of arc at 10,000
feet [Dedrick 1978] .

(8) Sextant Errors: There are seven sources of error in
the modern sounding sextant - ^ adjustable and 3 nonadjust-
able [Bowditch 1977] • One nonadjustable error is called
prismatic error and results from the two faces of the mirrors
not being parallel. The other two nonadjustable errors are
graduation errors (due to the arc or micrometer being
improperly cut) and centering error (due to the index arm
not pivoting at the exact center of curvature) . These are
usually combined into one error called instrument error for
which the manufacturer provides a correction table. The

adjustable errors are those resulting from nonperpendic-
ularity of (1) the frame and the index mirror, and (2) the
frame and horizon glass (side error) and the lack of
parallelism between, (3) the index mirror and horizon glass
at zero setting (index error), and (4) the telescope to the
frame (collimation error). Bowditch (1977) explains each
of these errors in detail and methods of adjustment to
minimize them.

APPENDIX B: THEODOLITE INTERSECTION POSITION ERROR

The best estimate for each sextant angle observed aboard
ACANIA was derived from the corresponding T-2 theodolite
determined position. Errors in these positions caused
inaccuracies in the best estimates of the angles. The
magnitude of these positional errors was dependent on the
angular resolution of the theodolite and the distance from
the theodolite to the position. A well-trained observer
using a T-2 theodolite on a stationary target during day-
light hours can measure an angle within -2.5" ninety percent
of the time when sixteen plate settings are used [Cervarich
1966]. This yields a standard deviation of 1.5"' However,
for only one observation on a moving target such as ACANIA,
a larger value must be used. Heinzen (1977) states that the
angular error in measuring azimuths for hydrographic vessel
positioning is -36 seconds. He does not present the method
used to derive this value, what probability is associated
with it or to what instrument it applies. Experience
indicates that this value is quite large, but to derive the
largest expected error in theodolite positioning of a
moving target it was assumed that the standard deviation of
the T-2 theodolite measurements was 36 seconds. A more
reasonable value of 20 seconds was also used for comparison.

The theodolite positioning error had varying effects on
the computed best estimates used for the sextant angles. If
displacement was along the circular line of position deter-
mined by the ship and the two signals then the error in the
best estimate was zero. Displacement normal to this line
resulted in the maximum error. The maximum errors were
determined for three points chosen near the extremes of the
work area and are summarized in Tables B-1 and B-2 . The
derivation of the values in these tables follows.

The root mean square error (d^ms) o^ 3- position is the
square root of the sum of the squares of the standard
deviations along the major and minor axes of a probability

ellipse. This is given by the equation 1 d-^i^s ~\l^t "^ ^J
The values a^ and Oy are not the same as the standard
deviations of the errors in the lines of position which are
given by o^ and 02- However, for two independent lines of
position, they are related by the following two equations
from Bowditch (1977):

ol = ^- I o^ + 0^ + \ko^ + Q^) - 4sin26a^a2']

X 2sin^6 1 2 V 1 ^ 12

y 2sin^6 1 2 V 1 2 12

6 is the angle of cut or angle of intersection of the two
lines of position. The result of substituting these
equations into the formula for root mean square error is

sm 6 V 1 2

For an azimuthal line of position the standard deviation
of the error is of the form a i = r j_ sin a for small values
of a . rj_ is the range from the point to the theodolite
station and a is the angular resolution or the standard
deviation of the angular measurement. By substitution the
root mean square error for azimuthal systems is finally

given by 1 drms = 5^^

^f!

^ + r^ . The probability

associated with root mean square error is not constant but
varies with the relationship between a^ and Oy. The
probabilities in Table B-1 that result from the three values
of Qy/ a were derived from Bowditch (1977) •

The errors in the best estimates of the angles ( 9i,
0 2 and 9 3) in Table B-2 were derived by contouring the
errors which resulted from shifting the theodolite deter-
mined positions + 1 meter in latitude and longitude. An

example is shown below in Figure B-1 .

-!• 0' +1'
/.18'

^^1.53'

+ 1 .71 •

Figure B-1 . Angular Error at Position 1 Due to 2 d^ms

Theodolite Positioning Error of 0.59^ Meters

POSITION

Latitude (N) 36
Longitude (W) 121

O37'09.825"
°53'30.088"

36036'^8.222"
121°53'03.28^"

36O37'07.210"
121°52'31.133"

106^m

1933m

2520m

1171m

715m

1701m

68.60

72.^0

^7.1°

20"

.103m

.187m

.2li,k-m

.ll^m

.069m

.165m

.137m

.198m

.37^m

.092m

.069m

.1^7m

V^x

.674

.3^8

.393

^ ^rms

.165m

.210m

.^02m

Prob. (1

^rms^

6i^.3^

67.^?^

67. 2f.

2 ^rms

.330m

.^20m

.80^m

Prob. (2

^rms)

97. 6f.

96. 3f-

96. 5?^

36"

.186m

.337m

.^^Om

.20^1-111

.125m

.297m

.2^6in

.356m

.675m

.166m

.12i^^m

.26iJ'm

^y/^x

.67^

.3^9

.392

^ ^rms

.297m

.377m

.725m

Prob. (1

^^rms)

6^.3?^

67.^%

67.2^

2 ^rms

.59^m

.75^m

1.^50m

Prob. (2

^rms)

97. 6f.

96. 3f-

96.5f«

TABLE B-:

1 . Theo

dolite Posi'

tioning Errors

at Three

Locations

POSITION

Latitude (N) 36^37 '09.825" 36036*^8 .222" 36O37'07 .210"
Longitude (W) 121°53'30 .088" 121O53'03 .28^^-" 121°52 '31 .I33"

20"

2 ^rms

.330m

.^20m

.804m

Q-^ error

0.5'

1.9-

1.2'

1.5'

2.1'

1.2'

0o error

1.7'

2.2'

1.3'

36"

2 "^rms

.59km

.75^m

.450m

0.9'

3.5'

2.1'

2.7'

3.8'

2.1'

©o error

3.O'

3.9'

2.3'

TABLE B-2.

Maximum Errors in Angular Best Estimates at
Three Locations Due to Theodolite Positioning
Errors .

The origin in the above figure represents position 1.
The grid around the origin are 1 meter shifts in latitude
and longitude. The value at each grid point is the differ-
ence in the angle 9^ at that point compared to the origin.
By drawing the appropriate d27ms circle (in this case 2 d^ms
= 0.59^ni) the corresponding angular error can be found.

It can be seen from Table B-2 that the angular errors
induced by the theodolite positioning errors are not
insignificant. However, the precise amount of error in the
best estimates is indeterminant since the angular resolution
of the T-2 theodolite angles is not known. The average
distance between the theodolite determined positions and
the corresponding mean sextant fixes for both cruises was
1.05m. In conclusion, the random errors in the horizontal
sextant angles measured aboard ACANIA were made larger by
the errors due to theodolite positioning. Hence, the values
of standard deviation for the cruise data are the maximum
expected errors for sextant angles measured at sea.

APPENDIX C: DATA SET STATISTICS

The coding system used for the data sets is shown
below. All values shown are in minutes of arc.

(1) Data Origin:

I - Cruise I

II - Cruise II

W - Wharf Number 2
B - Beach Lab

(2) Use of Scopes on Sextants:
N - no scopes

S - with scopes

(3) Observer Number (experience level decreases as

this number increases):

1 through 6
(^) Angle Designator:

Q I - 110/205

^2 - 205/202

03 - 205/107

05 - 110/302
(5) Repeated Data Set (if necessary):

(1) - first set

(2) - second set

TOTAL INl-Gj 1^2-Qi IN3-9 1 IN^-® i INf-®! IN6-3i

(32)

e +0.09 -0.83 -1.16 +0.36 -0.^^ +6.37

a 2.25 2.63 2.62 3.33 2.34 3.59

TOTAL INl-0i(2) IN2-Q2 IN3-®-:!
(10) ^

e +0.08 +0.88 +1.26

^ 3.8^ 2.55 3.44

TOTAL IISl-©! IIS2-9i IIS3-®! IIS^-Qi IIS5-®1 IIS6-©1
(32)

e +0.93 +1.31 +1.85 +1.78 +0.88 +2.18

a 1.28 1.76 1.25 1.61 1.67 1.28

TOTAL IISl-0i(2) IIS2-Q2 IIS3-®? llSk-'^^ IIS5-®l(2) IIS6-02
(32) ^ ^

E +2.if0 -0.2^ +1.^1-8 -0.^5 +1.74 -0.73

a 1.58 1.41 1.67 1.77 1.58 1.05

TABLE C-1 . Cruise I and Cruise II Data.

CLOCKWISE

MICROMETER (15) WNl-9Zj, WN2-eij. WN3-0Zj. WN^-Oi|, mS-Ql^, WN6-ei|

e +2.88 +0.86 -0.26 -3-38 +O.63 +0.08

a 1.25 0.92 1.12 1.15 0.47 0.80

COUNTERCLOCKWISE
MICROMETER (I5)

e +2. 5k +1.88 -0.9^ -2.6^ +0.20 -1.26

a 0.96 0.78 1.38 0.64 1.17 0.82

TOTAL (30)
i

CLOCKWISE

MICROMETER (15) WNI-O5 WN2-05 WN3-05 WN^-O^ WN5-05 WN6-05

i +^,.^3 +1.79 +1.77 +2.35 -0.51 +0.92

a 0.85 1.10 1.10 0.96 1.12 0.7^1-

+2.71

+1.37

-0.60

-3.01

+0.42

-0.59

1.11

0.98

1.28

0.99

0.90

1.05

-0.34

+ 1.02

-0.68

+0.7^^

-0.43

-1.34

COUNTERCLOCKWISE
MICROMETER (15)

e
/\

+2.62 +3.04 +0.48 +2.04 -0.90 +2.78
1.15 0.71 0.69 0.91 0.48 1.06

TOTAL (30)

e +3.52 +2.42 +1.13 +2.19 -0.70 +1.85

a 1.35 1.11 1.12 0.93 0.87 1.31

Ae -1.81 +1.25 -1.29 -0.31 -0.40 +1.86

TABLE C-2. Wharf Number 2 Data - Without Scopes

CLOCKWISE

MICROMETER (15) WSl-Oi^ WS2-0Z| WS3-GZ| WS^-Oz^, ]{iS5-Ql^ WS6-0i|,

e +1.04 +0.58 -0.43 +0.10 +0.83 +0.44

d 0.25 0.23 0.33 0.51 0.51 0.48

COUNTERCLOCKWISE

MICROMETER (I5)

i +0.66 +0.41 -0.62 -I.I6 +0.14 -O.O3

a 0.27 0.21 0.32 0.44 0.31 0.57

TOTAL (30)

e +0.85 +0.50 -0.52 -0.53 +0.49 +0.21

a 0.32 0.23 0.33 0.79 0.55 0.57

Ae -0.38 -0.17 -0.19 -1.26 -0.69 -0.47

CLOCKWISE

MICROMETER (15) WSI-O5 WS2-O5 WS3-05 WS4-05 WS5-O5 WS6-0-

£ +0.62 -0.37 +0.33 +0.04 +0.10 +0.66

a 0.18 0.24 0.14 0.42 0.28 0.46

COUNTERCLOCKWISE
MICROMETER (15)

+0.44
0.30

+0.03
0.32

+0.10

0.25

-0.25
0.23

-0.36
0.22

+ 1.34
0.45

TOTAL (30)

i
a

+0.53

0.26

-0.18

-0.17

0.35

+0.40

+0.21

0.23

-0.23

-0.10

0.36

-0.29

-0.13

0.34

-0.46

+ 1.00

0.57
+ 0.67

TABLE C-3. Wharf Number 2 Data - With Scopes

CLOCKWISE SEXTANT #2972

MICROMETER (15) BSl(l) BS2(1) BS3(1) BS^J-Cl) BS5(1) BS6(1)

i -0.^1-1 -0.59 -0.77 -0.35 -0.28 -0.91

a 0.28 0.25 0.21 0.25 0.2-4- O.kk

COUNTERCLOCKWISE
MICROMETER (15)

e
a

-0.59 -0.77 -0.9^ -0.56 -0.71 -0.58
0.33 0.1^ O.lii- 0.3^ 0.30 0.^2

TOTAL (30)

CLOCKWISE SEXTANT #2982

MICROMETER (15) BS1(2) BS2(2) BS3(2) BS^(2) BS5(2) BS6(2)

-0.50

-0.68

-0.85

-0.i^6

-0.50

-0.75

0.32

0.21

0.20

0.31

0.3^

0.14-5

-0.18

-0.17

-0.21

-0.^3

+0.33

-0.28 -0.0^1- -0.25 +0.09 -0.09 -0.67
0.36 0.23 0.28 0.36 0.28 0.^5

COUNTERCLOCKWISE
MICROMETER (15)

e -0.59 -0.14-7 -0.63 -0.59 -0.55 -0.39

a 0.34 0.2i^ 0.16 0.35 0.25 0.34

TOTAL (30)

e
a

-0.^3

-0.25

-o.kk

-0.25

-0.32

-0.53

0.38

0.32

0.30

0.49

0.35

0.i^2

-0.31

+0.01

-0.38

-0.68

-0.^^-6

+ 0.2i^

TABLE C-4. Index Correction Differences

CLOCKWISE SEXTANT #3003

MICROMETER (15) BS1(3) BS2(3) BS3(3) BS'4-(3) BS5(3) BS6(3)

i -0.33 -0.19 -0.'4'3 +0.39 -0.23 -0.63

a 0.31 0.20 0.23 0.39 0.30 0.63

COUNTERCLOCKWISE
MICROMETER (15)

e
a

-0.^5
0.37

-0.57
0.22

-0.b5
0.22

-0.57
0.30

-0.78
O.ij-O

-0.6^
0.50

TOTAL (30)

e
a

-0.39

0.3^

-0.12

-0.38

0.28

-0.38

-0.6^'

0.31

-0.^1

-0.09

0.60

-0.96

-0.50

0.^5

-0.55

-0.63

0.58

-0.01

TABLE C-^. Index Correction Differences (continued).

OBSERVER 1 2 3 -tf 5 6

WITH SCOPES

Sextant

#2972 -0.5 +0.1 +1.0 -1.0

-0.2 -1.0

-0.^■ -0.6

-0.5

#2982 -0.6 -0.5 -0.5 -0.5

-0.2 -0.6

-0.3 -0.2
-1.0

#3003 -0.6 0.0 — +o.k' 0.0 +0.3

-0.5 -0.5

-0.5 -0.6

-0.-4- -0.5

-0.7 -0.5

NO SCOPES

Sextant

#2972 +0.5 -0.5 +0.5 -0.2 -1.0

-0.5 -1.2 -1.0 -O.ij-

#2982 : -O.l^ -1.0 -1.0 +0.6 +0.8

-1.0 -0.8 +1.0

-1.0

#3003 -0.5 +0.^1- -0.3 +0.2

-0.5 -1.0

-0.4 -0.5

-0.6 -0.6

-0.4

TABLE C-5. Abstract of Index Correctors From
Cruise and Wharf Number 2 Data.

BIBLIOGRAPHY

Admiralty Manual of Hydrographic Surveying, v. 1,
p. 297-307, Hydrographer of the Navy, London, I965.

Army Engineer Geodesy Intelligence and Mapping Research
and Development Agency Report Number 3O-TR, Practical
Field Accuracy Limits for a Wild T-2 Theodolite, by
P.J. Cervarich II, August I966.

Blair, C., Coastal/Offshore Positioning by Optical Methods,
paper presented at American Society of Civil Engineers
Convention, Boston, Massachusetts, 2 April 1979-

Bodnar, A.N., NOAA - Pacific Marine Center, Seattle,
private communications, December 1979 and June I98O.

Bowditch, N., American Practical Navigator, v. 1, Defense
Mapping Agency Hydrographic Center, 1977'

Cotter, C.H., "A Brief History of the Method of Fixing by
Horizontal Angles", Journal of Navigation, v. 25, no. k,
p. 528-53^, October 1972.

Dedrick, K., Analysis of the Three-Point Fix, Report No.
77-3, Sacramento, June 1978.

Goodwin, E.M. and Kemp, J. F., "Accuracy Contours for
Horizontal Angle Position Lines" , Journal of Navigation,
V. 26, no. ^, p. 481-^85, October 1973-

Gossett, F.R., Manual of Geodetic Triangulation, Coast and
Geodetic Survey Special Publication No. 2^7, p. 13^-136,
U.S. Department of Commerce, 1971*

Heinzen, M.R., Hydrographic Surveys; Geodetic Control
Criteria, Master's Thesis, Cornell University, Ithaca,
New York, 1977-

Ingham, A.E., Hydrography for the Surveyor and Engineer,
p. 21-25, John Wiley & Sons, 197^-

Ingham, A.E., Sea Surveying, v. 1 and 2, p. 71-90, John
Wiley & Sons, 1975-

May, W.E., A History of Marine Navigation. 1st ed., p. 1^0,
W.W. Norton & Company, 1973-

Mueller, I.I. and Ramsayer, K.H., Introduction to Surveying,
p. 10-11, Frederick Ungar Publishing Co., 1979-

Umbach, M.J., Hydrographic Manual, ^th ed., U.S. Department
of Commerce, National Oceanic and Atmospheric Adminis-
tration, 1976.

University of New Brunswick-Department of Surveying
Engineering-Lecture Note No. 45. Hydrographic Surveying I,
by D.B. Thomson and D.E. Wells, p. 65, November 1977-

U.S. Coast Guard, COMDTINST M I65OO.I, Aids to Navigation
Manual, Positioning, p. 1-C, I978.

U.S. Coast Guard Report No. CC-D-l06-7^. An Analysis of the
Positioning Accuracy of Horizontal Sextant Angles, by
J.T. Tozzi and H.E. Millan, June 197^-

INITIAL DISTRIBUTION LIST

No. Copies

1 . Defense Technical Information Center
Cameron Station

Alexandria, VA 2231^1-

2. Library, Code 0142
Naval Postgraduate School
Monterey, CA 939^0

3 • Chairman

Department of Oceanography - Code 68
Naval Postgraduate School
Monterey, CA 939^0

k . Chairman

Department of Meteorology - Code 63
Naval Postgraduate School
Monterey, CA 939^0

5. LCDR Dudley Leath

Naval Postgraduate School - Code 68Lh
Monterey, CA 939^0

6. CDR Donald E. Nortrup
Commanding Officer
NOAA Ship Peirce

i^39 West York St.
Norfolk, VA 235IO

7. LCDR Gerald B. Mills

Naval Postgraduate School - Code 68Mi
Monterey, CA 939^0

8 . Director

Naval Oceanography Division
Navy Observatory

34th and Massachusetts Avenue NW
Washington, D.C 2039O

9- Commander

Naval Oceanography Command

NSTL Station

Bay St. Louis, MS 39529

10. Commanding Officer

Naval Oceanographic Office

NSTL Station

Bay St. Louis, MS 39529

11. Commanding Officer

Naval Ocean Research and Development

Activity
NSTL Station
Bay St. Louis, MS 39529

12. Director (Code PPH)
Defense Mapping Agency

Bldg. 56, U.S. Naval Observatory
Washington, D.C 20305

13. Director (Code HO)

Defense Mapping Agency Hydrographic

Topographic Center
6500 Brookes Lane
Washington, D.C 203I5

1^. Director (Code PSD-MC)
Defense Mapping School
Ft. Belvoir, VA 22060

15- Director

National Ocean Survey (C)
National Oceanic and Atmospheric

Administration
Rockville, IVID 20852

16. Chief, Program Planning and Liaison

(NC-2)
National Oceanic and Atmospheric

Administration
Rockville, MD 20852

17. Chief, Marine Survey and Maps (C3)
National Oceanic and Atmospheric

Administration
Rockville, MD 20852

18. Director

Pacific Marine Center - NOAA
1801 Fairview Avenue East
Seattle, WA 98102

19 • Director

Atlantic Marine Center - NOAA
i^39 W. York Street
Norfolk, VA 235IO

20. Commanding Officer
Oceanographic Unit One
USNS BOWDITCH (T-AGS21)
Fleet Post Office

New York, NY 09501

21 . Commanding Officer
Oceanographic Unit Two
USNS BUTTON (T-AGS22)
Fleet Post Office

San Francisco, CA 9660I

22. Commanding Officer
Oceanographic Unit Three
USNS H. H. HESS (T-AGS38)
Fleet Post Office

San Francisco, CA 9660I

23. Commanding Officer
Oceanographic Unit Four
USNS CHAUVENET (T-AGS29)
FPO, San Francisco, CA 966OI

21^'. CDR R. A. Anawalt

Chairman, Oceanography Department
U.S. Naval Academy
Annapolis, MD 21ift)2

25. LT Kenneth Perrin
NOAA Ship Mt. Mitchell
^1-39 West York Street
Norfolk, VA 235IO

26. LCDR Donald Winter
105 Moreell Circle
Monterey, CA 939^0

27. Ms. Patricia Eaton

300 Glenwood Circle #133
Monterey, CA 939^0

28. LCDR Francisco Abreu
Institute Hidrografico
Rua Das Trinas , ^9
Lisbon - 2

Portugal

29. LTJG Luis Faria
Institute Hydrografico
Rua Das Trinas, ^9
Lisbon - 2

Portugal

30. LT Ali Kaplan
Bostanci Kbyu
Gonen - Balikesir
Turkey

esis

Thesis
c.l

190^24

Mills

Analysis of random
errors in horizontal
sextant angles.

♦ UQ^ 9Z S^Vvix.

2 723^9
2723> '
27235'

Thesis 190624

M5945 Mills

c.l Analysis of random

errors in horizontal

sextant angles.

Internet Archive Audio

Featured

Top

Images

Featured

Top

Software

Featured

Top

Books

Featured

Top

Video

Featured

Top

Mobile Apps

Browser Extensions

Archive-It Subscription

Save Page Now

Full text of "Analysis of random errors in horizontal sextant angles."

See other formats