NAVAL POSTGRADUATE SCHOOL

Monterey, California

THESIS

	ANALYSIS OF RANDOM ERRORS IN HORIZONTAL SEXTANT ANGLES
	by
	Gerald B.	Mills
	September	I98O
Thesis	Advisors:	D	. E D.	N W.	ortrup Leath

Approved for public release; distribution unlimited

T197453

SeCUMlTY CLASSIFICATION OW THIS VaRC fW»>mm Dmtm Bitfrm^)

REPORT DOCUMENTATION PAGE

nt^oAT NUMScn

READ INSTRUCTIONS BEFORE COMPLETING FORM

S. aOVT ACCIUIOM MO.

|. HCCIPiCmT'S CATALOG NUWaEff

« TITLE r«nOu*f((f«)

Analysis of Random Errors in Horizontal Sextant Angles

I. Tv^e ow «e^oi»T * pcrioo covcmeo

Masters Thesis September I98O

•. ^inroMMiNO ona. mi^out numsin

• . CONTHACT on CHANT NUMSCMC*)

7. AuTHonra;

Gerald B. Mills

to. PMOGHAM CLCMCNT. nnOjeCT, TASK AntA * WOHK UNIT NUMCCns

I. Hen^OHMINO OnOANlZATIQN NAME AI^O AOQAII$

Naval Postgraduate School Monterey, California 939^0

M CnNTWOLLINO Or^lCK NAME ANO ADOHCSS

Naval Postgraduate School Monterey, California 939^0

12. nCPONT DATE

September I98O

II. NUMSCn OF ^ACES

72

14. MONITORING AGENCY NAME A AOOHESS/K rflffarwit trmm Cwiffolllfii Otlieu)

Naval Postgraduate School Monterey, California 939^0

It. ttCUniTV CLASS, (ol iHI» riporl)

Unclassified

Mm. OKCUACllFlCATION/DOWNGnAOiNG tCHEOULC

l«. OlSTAtauTlON STATEMENT (ol (Ma Ra»«rO

Approved for public release; distribution unlimited

17. OtSTNiauTlON STATEMENT fal ihm «Aarraef anlara^ in •l«e* 30, II dllttmtt hmm Km^tt)

It, SU»I>LEMENTAI«Y NOTES

It. KEY WOAOt CContlnua am r^rtmm at4» II nmemmamrr «n4 <tf«t«tliy trp Mack nuailMr)

Hydrographic Surveying; Sextant; Three-point Sextant Fix; Horizontal Sextant Angles; Horizontal Positioning

20. ABSTMACT (Conllmf an

tmrr m*4 Id—tiltr tr tl««* nuaiAac)

The three-point sextant fix has been used for the horizontal positioning of vessels in nearshore hydrographic surveys since ^775' However, this method has only recently been modeled mathem.atically . The accuracy of the three-point fix depends on the magnitude of the random and systematic errors in the angle measurements and the fix geometry. The random errors in horizontal sextant measurements were investigated by analyzing

DO I jAN*71 1*^73 EDITION OF 1 NOV •! IS OBtOLlTI

(Page 1) S/N ai03>OI«t AAOI

^^^iMMMMMtaib^^MtMi^Siai^kUUti^kM^j^t^m^austiL

over 1^00 angles, both at sea and on land. These random errors were found to vary with the clarity of the signals being observed, the stability of the vessel and the experience of the observer. The upper and lower bounds for one standard deviation were found to be about 2.7 and 0.6 minutes of arc respectively. In addition, angular differences due to the direction of rotation of the micrometer drum were examined as well as the variability in the determination of sextant index error.

DD Form 1473 P

S/N 0102-014-6601 itcu»«T» clami^icatiom o^ tmu ^•ocrw**" o«»» f-«»'«*»

Approved for public release; distribution unlimited

Analysis of Random Errors in Horizontal Sextant Angles

T^y

Gerald B. Mills Lieutenant Commander, NOAA B.A., Washington State University, I967

Submitted in partial fulfillment of the requirements for the degree of

MASTER OF SCIENCE IN OCEANOGRAPHY (HYDROGRAPHY)

from the

NAVAL POSTGRADUATE SCHOOL September I98O

c \

ABSTRACT

The three-point sextant fix has been used for the horizontal positioning of vessels in nearshore hydrographic surveys since 1775- However, this method has only recently been modeled mathematically to quantify the effects of various errors on fix accuracy. Positioning error in the three-point fix depends on the magnitude of the random and systematic errors in the angle measurements and the fix geometry. Random errors in horizontal sextant measurements were investigated by analyzing over 1^00 angular observa- tions, both at sea and on land. These errors were found to vary with the clarity of the signals being observed, the stability of the vessel and the experience of the observer. The upper and lower bounds for one standard deviation were found to be about 2.7 and 0.6 minutes of arc respec- tively. In addition, systematic errors resulting from angular differences due to the direction of rotation of the micrometer drum were examined as well as the variability in the determination of sextant index error.

TABLE OF CONTENTS

I. INTRODUCTION 9

A. ■ HISTORICAL BACKGROUND 9

B. THREE-POINT FIX METHOD AND POSITIONING ACCURACY. 11

C. OBJECTIVES 1^

II. DATA COLLECTION METHODS 19

A. CRUISE DATA 19^

B. STATIONARY DATA 25

C. INDEX ERROR AND INSTRUMENT ERROR DATA 26

III. DATA REDUCTION PROCEDURES 30

IV. RESEARCH RESULTS 34

A. ABILITY OF THE OBSERVER 3/I

B. DISTINCTNESS OF SIGNALS . 36

C. PLATFORM STABILITY 38

D. DIFFERENCES DUE TO DIRECTION OF ROTATION

OF THE MICROMETER DRUM -^0

E. INDEX ERROR DETERMINATION l^-l

F. CRUISE II SYSTEMATIC ERROR t^,^

V. CONCLUSIONS k6

APPENDIX A. BLUNDERS AND SYSTEMATIC ERRORS AFFECTING

THREE-POINT FIX POSITIONING ACCURACY .... ^9

APPENDIX B. THEODOLITE INTERSECTION POSITION ERROR . . . 51^

APPENDIX C. DATA SET STATISTICS 60

BIBLIOGRAPHY 6?

INITIAL DISTRIBUTION LIST 69

LIST OF TABLES

TABLE I - Previous Values for the Magnitude of

Random Errors 15

TABLE II - List of Stations 21

TABLE III - Data Set Reference Code 33

TABLE IV - Mean Differences Between Simultaneously

Measured Angles 35

TABLE V - Statistics for Inverse Distances Between Main

Three-Point Fixes and Check Fixes 3^

TABLE VI - Comparison of Cruise Data to Wharf

Number 2 Data 38

TABLE VII - Index Corrections from Thirty Observations. • ^2

TABLE VIII - Instrument Errors • ^5

TABLE IX - Summary of Ranges of Sample Standard

Deviations ^6

TABLE B-1 - Theodolite Positioning Errors at Three

Locations '51

TABLE B-2 - Maximum Errors in Angular Best Estimates at

Three Locations Due to Theodolite Positioning Errors 58

TABLE C-1 - Cruise I and Cruise II Data 61

TABLE C-2 - Wharf Number 2 Data - Without Scopes .... 62

TABLE C-3 - Wharf Number 2 Data - With Scopes 63

TABLE C-^ - Index Correction Differences ^^

TABLE C-5 - Abstract of Index Correctors from Cruise Data

and Wharf Number 2 Data 66

6

LIST OF FIGURES

Figure 1 . Three-Point Fix Geometry 12

Figure 2. Project Area and Station Locations 20

Figure 3* Template, Sextant and Tribrac -

Disassembled 28

Figure k. Template, Sextant and Tribrac -

Assembled 28

Figure 5- Data Distribution for ©i (Angle 110/205) -

Cruise I (top) vs. Cruise II (bottom) .... 37

Figure 6 . Distribution of Wharf Number 2 Data - Without

Scopes (top) vs. With Scopes (bottom) .... 39

Figure B-1 . Angular Error at Position 1 Due to 2 dj^ms

Theodolite Positioning Error of 0.59^ Meters . 56

ACKNOWLEDGEMENTS

I would like to express sincere gratitude to my classmates for collecting the data which made this study possible and to LCDR Donald E. Nortrup and LCDR Dudley W Leath, my advisors, for their suggestions and assistance I would especially like to thank my loving wife B.J. for typing the numerous versions of this thesis and for her understanding and patience throughout its preparation.

8

I. INTRODUCTION

A. HISTORICAL BACKGROUND

Instruments for measuring the altitude of the sun or stars have existed for over 2,000 years, but it was not until 1730 that the forerunner of the modern marine sextant appeared. Two inventors, John Hadley of England and Thomas Godfrey of Philadelphia, simultaneously designed instruments using one fixed mirror and one movable mirror to measure angles [May I963 ]. Both instruments allowed the movable mirror to rotate through an arc of ^5° » but due to their double mirror construction were able to measure angles up to 90°. Hence, they were called quadrants, although octants would have been a more proper name. These quadrants were not readily adopted by navigators and it was 1750 before Hadley 's quadrant was in general use aboard vessels of the East India Company [Cotter 1972] .

The need to measure angles greater than 90° prompted Captain Campbell R.N. in 1757 to suggest the enlargement of the arc of Hadley 's quadrant to 60° enabling the measurement of angles up to 120°. Hence, the name sextant from the Latin word sextans, "the sixth part" [Bowditch 1977] •

-t

Bowditch (1977) discusses the optics of this construction which is similar to the modern sextant.

Although the sextant was designed primarily for measuring vertical angles, it can be held on its side to measure horizontal angles as well. Rev. John Mitchell first suggested the use of the sextant for measuring horizontal angles in hydrographic surveying in February ^7^5 [Cotter 1972]. Mitchell's method of fixing a vessel consisted of intersecting the line of position derived from a horizontal angle between two known points'^ and a position line obtained from a compass bearing to one of the points. In l??^ » the first Hydrographer to the Admiralty, Alexander Dalrymple , suggested determining position by intersecting the lines of position derived from horizontal angles between three or more known points [Cotter I972]. This principle had been known by land surveyors since the early seventeenth century but its application aboard ship was delayed due to the lack of an accurate angle -measuring device .

The first application of Dalrymple 's suggestion was made by Murdoch Mackenzie II. He surveyed the channels off the Kent coast of England between l??^ and 1777 • Positions could be plotted quickly by using a device called a three- arm protractor or station pointer [ Admiralty Manual of Hydrographic Surveying 1965]' This method for positioning

2

A known point is a reference station whose geodetic

coordinates have been determined. Also called a horizontal control point.

10

is called the three-point fix or resection method and is used today worldwide for inshore hydrographic surveys.

There have been several changes in the equipment used in the three-point fix method over the last 200 years. An endless tangent screw for continuous tracking and a micro- meter drum for increased accuracy were added to the sextant during the twentieth century. The marine sextant is fitted with darkened shade glasses for observation of the sun. A specialized sounding sextant has been developed specifically for hydrographic surveying using lighter weight materials and more rugged construction [Ingham 1975 ] • In addition, the sounding sextant has a wide angle low magnification telescope and a micrometer graduated in minutes of arc . In recent years an electronic digital sounding sextant was developed to enter observed angles directly into a mini- computer aboard a vessel [Umbach 1976] . Despite these improvements in the sextant the most draiiiatic equipment change has been in the area of position plotting. Shipboard computers and automated plotters have largely replaced the three-arm protractor resulting in increased accuracy.

B. THREE-POINT FIX METHOD AND POSITIONING ACCURACY

Figure 1 illustrates the geometry of a three-point fix. The known stations are depicted by triangles and labelled A, B and C The vessel is located at point P and the observed sextant angles are given by the symbols 0i and Oz- The

11

u ■p o

B o

0)

o

X

■P

•H O

I

0) 0 f>4

Eh

0

•H

Pl4

12

angle Qi and the points A and B define a locus of points which is a circle through A, B and P. At any point on that position circle, or line of position, the angle between A and B is always e^. Similarly, there exists a position circle for points B and C and angle 02 • A third line of position through points A and C is defined utilizing the angle ( 0i + 62) •

As can be seen in Figure 1, the three lines of position intersect at point P. The accuracy of the position of P depends on the errors in the measured quantities used for position determination. Errors are classified into three types; blunders, systematic errors and random errors. Blunders are simply mistakes such as misreading a sextant or improperly identifying a signal.^ They are eliminated from the data by comparison with redundant or related observations and careful editing. Systematic errors follow some mathematical or physical laws and therefore have a fixed relation to a set of conditions. For example, elevation differences in observed signals cause systematic errors in sextant measurements. In this case the conditions (the elevation differences) can be measured and the correc- tions can be calculated and applied. However, the laws associated with systematic errors are not always known.

^ A signal is a natural or artificial object located at a survey station (known point) which is used as a sighting point for sextant measurements.

13

Systematic errors from unknown sources can be minimized by following sound measuring techniques and instrument calibration procedures. Random errors are inherent in all physical measurements and can not be removed from the data. Their effects must be estimated statistically. An example of random errors would be measuring a known 50° angle several times and obtaining sextant readings between 49°57.2' and 50^02. 8' .

Blunders and systematic errors affecting the accuracy of positions determined by the three-point fix method are discussed in Appendix A. The random errors in horizontal sextant measurements at sea are the subject of this paper.

C . OBJECTIVES

Until recently, there have been very few attempts to quantify the accuracy of sextant positions. Several formulas have been developed in the last decade to do this using the magnitude of the random errors as one of the parameters. Tozzi (197^-) developed a series of equations which included among other variables, the standard deviation of the random error in measuring angles. Thomson (1977) used the confidence intervals associated with the two angle measurements of a three-point fix. The equation in Bowditch (1977) includes the error in measurement of the horizontal angles. Heinzen (1977) used the mathematical notation for the standard deviation of the observed angles

Ik

in his development. Dedrick (I978) developed a formula for the area of the error ellipse about some position given the standard deviation of the sextant angle measurement error.

There seems to be no agreement among the various authors as to the magnitude of these random errors. The table below illustrates this.

Author		Year	Error
Goodwin		1973	6' 05.5"
Ingham		197^	+ 0.5'
Ingham		1975	+ 1 .0'
Thomson		1977	^'
U.S. Coast	Guard	1978	several minutes
Dedrick		1978	few to several minutes
Bodnar		1978	0.7' - 1.3'

TABLE I. Previous Values for the Magnitude of Random' Errors

The errors mentioned by Goodwin, Thomson, Dedrick and Bodnar specifically refer to the standard deviation of the random errors while Ingham and the U.S. Coast Guard do not. It should also be noted that Tozzi (197^) used Goodwin's standard deviation in his development and Heinzen (1978) used Ingham's 197^ value. All of the above errors are estimates arrived at through experience except those of Goodwin and Bodnar. Goodwin arrived at his result by calculating the standard deviation of 32 angular measure- ments between two well defined stations. He had 32 experienced navigating officers each measure the angle once

15

Bodnar's figures resulted from a one day experiment aboard the NOAA Ship DAVIDSON while moored at a pier at Lake Union in Seattle, Washington. Theodolite observations were made from the ship to three easily identified objects which were generally at the same elevation. Conditions of extreme vessel stability due to tight mooring lines, little wind and no tide enabled Bodnar to achieve agreement between successive theodolite observations of about 0.5'- The two angles (one of about 6^ and the other about 50°) were then measured JO times each by six officers. The means and standard deviations were calculated for each of the 12 data sets. All of the means were within 0.5' of the angles determined by the theodolite measurements. The individual standard deviations ranged from 0.?' to 1.5'' No cumula- tive statistics -were determined.

Dedrick (1978) attempted to use historical data to arrive at his estimate. During a survey of south San Francisco Bay between 1857 and 1858 it was common practice to take full rounds of angles at a station. That is, angles were measured between objects all around the horizon. He studied 17 rounds of angles each consisting of four to seven sextant angles. The disagreement between each round and 3^0° was generally between 10' and 20' of arc, but ranged from 2* to 55' of arc. He states that "this data would suggest that as an extreme upper limit, values of of a few to several minutes of arc might be appropriate."

16

The main objective of this study was to quantify the random errors in sextant angle measurements under varying conditions. The factors upon which these errors are depen- dent are: (1) the ability of the sextant observer,

(2) the visibility and distinctness of the signals, and

(3) the stability of the platform. The ability of the sextant observer was analyzed by comparing more experienced observers with those with less experience. All observations were made across water on clear, somewhat windy days so that horizontal refraction was at a minimum. The distinctness of the signals was altered by using telescopes on the sextants for some observations and not for others . Platform sta- bility was analyzed by measuring angles under three conditions: (1) vessel in moderate to heavy seas,

(2) vessel in calm seas, and (3) observer at a stationary point on a wharf.

During the course of data collection a few other questions arose which were related to random errors. Is there a difference in angle measurement if the micrometer is turned clockwise or counterclockwise? How much do individ- uals vary when determining index error on the same sextant? Are the manufacturer's stated instrument errors correct? Answers to these questions and the analysis of the random errors are discussed in later sections.

The scope of this paper does not include the positional accuracy of three-point sextant fixes. Tozzi (197^) »

17

Heinzen (1978) and especially Dedrick (1978) covered this subject in detail. No attempt was made to evaluate the random errors while a vessel was underway as it would be when running a hydrographic survey line. The data analysis in this thesis corresponds to the use of the three-point fix for calibrating or evaluating electronic positioning control

18

II. DATA COLLECTION METHODS

Raw data for this study consists of sextant angle measurements observed in the southern portion of Monterey Bay, California. This area and the horizontal control stations used during the project are shown in Figure 2. All of the station positions were determined by third-order methods or better by personnel of the National Ocean Survey. These stations, their positions, elevations and station numbers and the sources of this information are shown in Table II. The data was collected both at sea and on land and will therefore be discussed separately. All sextants were checked each day for adjustable errors and were found to be satisfactory.

A. CRUISE DATA

The 126-foot long research vessel ACANIA was used as the observation platform for the anglemen (sextant observers) during the two data collection cruises. The position of the ship was determined for every sextant observation by the standard theodolite intersection method described by Umbach (1976). Briefly, this method consists of occupying two horizontal control stations with theodolites. Each theodolite operator measures the angle between another known point and the object to be located.

19

I

121* 54- W

I 121* 53-W

AMERICAN CAN

CO. STACK A

(202) ^-^

KMBY MAST (203)

MONTEREY

2^26-U

MONTEREY CO. DISK (301)

'MONTEREY HARBOR LT. 6 (205)

FT. ORD SILVER

WATER TANK (206)

use: mon ^

■ (110)

NPGS TOWER A (302)

WW'^M CRUISE I ^^^ CRUISE n

Figure 2. Project Area ajid Station Locations

20

en

	VO	VO
5	O	CO
Q)	»-H	T-1

Cv.'

>)	>5		>5
fil	^		^
rH	iH		iH
0)	<u		(U
1	S	CM	1
00	en	\o	C/0
o	o	o	o
s	2	«-<	2

o w

> w

c^

cvj

o

r^ vPi \o

S^	w^	C^	o	o	CO	CO	v>,	00	^
	ON	«-l	o	u^	c--	o\	^J^	CM	O
	f^	vo	NO	CJN	NO	ON	o	NO	o
w		•		•		•	•	•	•
a	o	CO	CM	n	^	c^	ON	t-l	>J^
:r)	«-l	♦H	r^	VA	vr>	CM	1H	1-t	r^
Eh		-					-	-	.
M	^	r^	CM	C^	C^	C^	C^	^	OJ
C3	>J-^	^	vn	^	VA	u^	u^	u^	VA
2	o	o	o	O	O	o	o	O	o
O	<-i	tH	tH	1-1	«-i	r-t	♦H	t-(	«-l
h-J	CM	CM	CM	CM	CM	CM	CM	CM	eg T-l

CM

o\	c^	CM	C3N
^	^	CM	CO
00	NO	CM	CN-

NO

^

NO

00

NO

w	.	.	•	.	.	.	.	.	.
Q	^A	CO	c^	NO	NO	CM	o	CO	^
:d	O	o	u^	u^	VA	r^	O-N	1-1	o
Eh				-		-	-	.
M	o-	NO	u-\	^-	NO	NO	NO	c^	NO
E-i	C^	c^	r^	r^	c^	o->	c^	c^	f^
S	o	o	o	o	O	o	o	o	o
NO	NO	NO	NO	NO	NO	NO	NO	NO
	0^	r^	c^ E-* C/3 O	0^	C^	0^	r^	r^	c^
	«		s	EH
	o		s
	<:		o	a:			NO
	EH		ir>	w
i	W		W	Eh		i^	Eh k5
<	•		o	s		00
S	o		Ph			M	CC
	o		2	K	CM	Q	O
s				W	NO		PQ
o	2		w	>	C3N	•	CC
H	<		!=)	h^	tH	o	<	CM
Eh	o			M		o	X	0^
<			M	t/J	Eh			CJN
Eh	2		^		in	>H	J^	vH
W	<:		2	Q	<j	w	W		2
	o		O	CC	g	K	cc	^	o
	M		s	O		W	w	w	s
	OS	Eh			>H	Eh	Eh	CO
		^	jrl	•	PQ	2	2	t/3	w
	s	O	W	&^	S	O	O	p	00
	<:	PQ	Q	Cx.	t:d	s	s	g	p
Z
o cc
M W	CM	o	CM	NO	O^	»-l	vo	c^	o
EH PQ	o	U>1	o	o	o	o	o	o	«-l
< s	CM	c^	c^	CM	CM	c^	CM	tH	»H
EH&
W 2

0 w

•H

x:

O

13 0)

-P CS

o

•H

C

•H

W W

o

rH

C

3

CM

«H NO

r^

a

E o

u

£

c a> cd «

01

C o

•H

+>

a +j w

o

•H CO •H

9

E-»

21

Since the azimuth can be computed between any two known points, the azimuth to the unknown point can be determined. Therefore, the theodolite measurements from each station produce lines of position, the intersection of which is the location of the unknown point. To ensure the correctness of the beginning azimuth, a check azimuth to a second known point is usually observed.

Stations MUSSEL (10?) and MONTEREY CO. DISK (301) were occupied with Wild T-2 theodolites on both cruises. USE MON (110) was observed to obtain the initial for MUSSEL (107) and MONTEREY HARBOR LT 6 (205) was the azimuth check. The theodolite observer at MONTEREY CO. DISK (301) initialled on IVIUSSEL (10?) and sighted on USE MON (HO) as the azimuth check. (Subsequent references to these and other stations will only be by the station numbers indicated in Table II.) Both theodolite operators attempted to use the center of the three anglemen as a target. However, since this was not easily visible at all times, a bright orange float, 2 feet in diameter, was tied to the ship's rail beside the observers.

The sextant observers consisted of six individuals with three levels of experience in measuring angles with sex- tants. Two anglemen (1 and 2) had extensive horizontal angle measurement experience. Two others (3 and ^) had extensive experience with measuring vertical angles with a sextant, but none with measuring horizontal angles. The

22

final two (5 and 6) had no prior sextant experience. The six observers were divided into two groups of three. The more experienced group consisted of the first two observers and anglemen number 3- Each group of three measured angles simultaneously from the aft portion of the upper deck of ACANIA. The observers stayed within one meter of each other to minimize the effects of eccentricity.

Observations from the ship were coordinated with those ashore by using three portable CB radios. To reduce the positioning errors caused by theodolite mispointings the captain of ACANIA attempted to stay within a small area about ^50 meters NNW of the Coast Guard pier. This position minimized the distance to the two theodolite locations and gave an intersection angle of near 90°. Unfortunately, weather conditions made it impossible to maintain station. Figure 2 shows the areas of operations for both cruises.

Cruise I took place under adverse weather conditions. Winds were 20-25 knots from the WNW with seas of approximately 5 to 8 feet. ACANIA, with its 126-foot length and 22-foot beam, provided a very unstable platform. By not using scopes on the sextants under these conditions of poor platform stability, the largest value for the standard deviation of the random errors was obtained.

Each angleman measured the angle between signals 110 and 205 a total of 32 times. In addition, the more experienced group measured the angles required to compute ten pairs of

23

three-point fixes. Each pair of fixes was derived from a left angle 0i(llO/2O5), a right angle 02(205/202) and a right check angle 0^ (205/10?). The fix determined from angles Qi and ©2 was designated the main fix and the fix computed from 0^ and Q-^ was called the check fix. The distance from the main fix to its corresponding check fix was determined and is called the inverse distance. Blunders in the horizontal angles can "be detected by analyzing the size of these inverse distances. If the values are less than five meters, then the National Ocean Survey considers that no "blunders have been made. For the ten pairs of fixes from Cruise I the mean inverse distance was 1.218 meters with values ranging from 0.003 to 2.390 meters (standard deviation = 0.80 meters). Planned operations to collect additional angle data for three-point fixes were cancelled due to worsening weather conditions and failing radios. A total of 222 sextant angles were collected for analysis.

The weather conditions for Cruise II were much better than those for Cruise I. The wind was from the southwest at 5 to 10 knots with no appreciable seas. However, long period swells of 1 to 2 feet caused some vessel motion. Scopes were used on the sextants in an attempt to determine a reasonable least value for the standard deviation of the random errors at sea. It was planned to also collect data with no scopes, but a heavy rainstorm reduced visibility to less than 1,000 meters.

24

On this cruise each angleman took 32 measurements of the angle 110/205 • As in Cruise I, angles were observed to compute pairs of three-point fixes. The same signals were used for both cruises. Each group measured a total of 96 angles resulting in 32 pairs of fixes. The mean inverse distance was 0.7^9 meters with values ranging from O.Oll to 2.555 meters (standard deviation = 0.61 meters). The total number of angles measured for Cruise II was 38'^*

B. STATIONARY DATA

All of the data collected at sea was subject to possible errors from the observers not being in the exact same location and from the theodolite intersection method. These errors are discussed in Appendices A and B respectively. The random errors were also influenced by platform stability. To evaluate the effects of this factor the observers measured a series of angles from station 350 on Wharf Number 2 at the Monterey Harbor. A T-2 theodolite was used at 350 to measure the horizontal angles between 110 and 206 and between HO and 302. These angles were measured according to the third-order specifications prescribed by Umbach (1976) which require four measurements with different plate settings, all within 5 seconds of the mean.

Each of the 6 anglemen stood directly above 350 and measured both angles 30 times with scopes and 30 times without scopes. In addition, each 30 observation set was

25

divided such that 15 angles were measured with the micrometer drum being turned clockwise or decreasing in value and 15 angles were measured with counterclockwise motion. Since the observed stations were not at the same elevation as 350 i the observed angles were not horizontal angles. The equation from Umbach (1976) for converting inclined angles to horizontal angles and vice versa is given in Appendix A. The angular elevations of each object at 350 were measured with a T-2 theodolite. Hence, the horizontal angles 206/110 and 110/302 were converted to the inclined angles that were observed by the sextants. The errors were then calculated for each of the 720 sextant angles.

C. INDEX ERROR AND INSTRUMENT ERROR DATA

Tests were conducted to determine if any systematic errors were unaccounted for in the previous data. Each individual determined index error for his sextant every time he made a set of observations. However, there were larger than expected differences of index error between individuals using the same sextant. Individual determina- tions also varied from day to day. Therefore, index errors were studied by having each angleman make 30 measurements of index error with each of the three sextants. The procedure consisted of holding the sextant vertically, observing the sea horizon and bringing the direct and

26

reflected images into coincidence. As before, 15 measurements were made with a clockwise micrometer drum movement and 15 with a counterclockwise movement. A total of 5^0 index error measurements were analyzed.

Instrument error was also analyzed for each sextant. Recall that instrument error consists of graduation error and centering error. This is determined for each instru- ment by the manufacturer and the results are posted inside the sextant case. Each of the three sextants used on this project had posted correctors of 0.0 minutes for each 15° increment of arc. To check some of these values a method suggested by CDR . James Wintermyre of the NCAA Pacific Marine Center's Operation Division in Seattle, Washington, was used. He reasoned that to properly "calibrate" a sextant or determine its instrument error for various angles, accurate theodolite angles must first be observed between objects. The sextant could then be used to measure the same angles. To maximize the accuracy of this -calibra- tion the sextant must be perfectly horizontal and over the identical point from which the theodolite angles had been measured. To accomplish this he designed a sextant template that would mount in a T-2 theodolite tri-brac and would accommodate three different makes of sextants. The template, tri-brac and a sextant are shown in Figures 3 and ^.

Initially a temporary reference station was established from which several objects at the same elevation could be

27

Figure 3- Template, Sextant and Tribrac - Disassembled

Figure k. Template, Sextant and Tribrac - Assembled

28

sighted on. The objects were the southwest edge of a chimney, a flagpole on Wharf Number 2 and the southern edge of the AMERICAN CAN CO. STACK (202). Their elevations were determined to be the same as the theodolite by setting 90° on the vertical circle. The angles between the objects were measured with the theodolite, again to third-order specifications. Unfortunately, the theodolite did not have a removable tri-brac . Hence the template was mounted on another tri-brac which was centered over the station. The "bull's-eye" level bubble of this new tri-brac was out of adjustment. But since the proper vertical location on each target was well determined by the theodolite, the tri-brac was adjusted to allow sighting on these points with the sextant. Each angle was measured 30 times with each sextant - 15 times with a clockwise micrometer movement and 15 times in a counterclockwise direction. Thus, a total of 180 angles were observed for analysis.

29

III. DATA REDUCTION PROCEDURES

All computer work for this project was done on the IBM 360/67 system at the Naval Postgraduate School's W.R. Church Computer Center. All programs were written in FORTRAN IV. Program UCOMPS was used to determine positions from three-point sextant fixes and T-2 theodolite inter- sections. It was provided by the National Ocean Survey - NOAA, Rockville, Maryland. Program INVERS was utilized to compute the lengths and azimuths of lines between known points. This program was obtained from the National Geodetic Survey - NOAA, Rockville, Maryland. Library routine HISTF was used for the statistical analysis of all data for this project.

Error analysis requires that the best estimate of a measured quantity be determined. Usually this value is the mean of a number of measurements of a particular angle or distance. However, the sextant angles observed during the two cruises could not be treated in this manner since ACANIA was always moving. Thus each angular measurement was a unique and unrepeatable observation. Therefore, the best estimate of each sextant angle observed aboard ACANIA was derived utilizing the theodolite determined ship positions. Inverses were computed from each position to the horizontal control stations upon which the sextant observations were

30

made (station numbers 110, 205, 202 and 107). The difference in forward azimuth to any two stations from the ship's position was chosen the best estimate of that angle. Likewise, the best estimates for the sextant angles observed at Wharf Number 2 were the inclined angles derived from the T-2 theodolite measurements there (see page 26).

Errors ( e ) were calculated for each observed sextant angle (x) by subtracting the best estimate of the angle (X) . This is expressed mathematically as e j_ = Xj_ - Xj^ where i is the number of the observation. The mean ( i ) of the

errors for each data set was then calculated as follows:

n e= .Z.z-/n where n is the total number of observations. If

1=1 T

the errors in the sextant angles are normally distributed and totally random, then by definition e must be equal or nearly equal to zero. If e does not equal zero for each data set, then sampling errors and/or undetermined systematic errors were present. The bounds of sampling errors for a given probability are directly proportional to the sample standard deviation of the data set and inversely proportional to the square root of the number of errors analyzed.

The sampling and systematic errors must be eliminated when calculating the sample standard deviation ( a ) of the

k

The errors determined for each observer under each set of conditions are referred to as a data set.

31

random errors. This was done for each data set by using

I n 9 the equation a = .[ Z (e. -e) ]/(n-l)

\| i = l

It can be shown that a is a measure of precision and e is a measure of accuracy .

The mean of the errors ( i ) and the sample deviation ( a ) for each data set and for some selected combination of data sets were calculated. These results are discussed in the next chapter and are summarized in Appendix C . Graphs of the distribution of the errors for some of the combined data sets are also shown. These graphs called frequency polygons are formed by connecting the mid-points of the tops of the bars in the histograms of the data.

A coding system was devised to simplify references to various data sets or data set combinations. The code consists of four or five characters and is shown in Table III.

The fifth character of the code is used for the cruise data if angle Qi was reobserved when collecting data for three-point fixes. Hence, data set INl - 0;[(2) would refer to the second data set collected by observer 1 while mea-- suring angle Q^ with no scope on Cruise I. Combinations of data sets are referred to by the part of the data that is common to all sets. For example, combination data set IN - Qi would refer to all no scope observations of angle 01 on Cruise I.

32

(1) Data Origin:

I - Cruise I

II - Cruise II

W - Wharf Number 2 (Stationary) B - Beach Lab (Index Error Data)

(2) Use of Scopes on Sextants: N - no scopes

S - with scopes

(3) Observer Number (experience level decreases as

this number increases:

1 through 6 (^) Angle Designator: % - 110/205 ®2 - 205/202 03 - 205/107 % - 206/110

e^ - 110/302

(5) Repeated Data Set (if necessary):

(1) first set

(2) second set

TABLE III. Data Set Reference Code

33

IV. RESEARCH RESULTS

The factors affecting the random errors in horizontal sextant measurements are the ability of the sextant observer, visibility and distinctness of the signals and the stability of the platform. The effects of these factors are not independent, and it is therefore impossible to isolate the contribution of each error source. In addition, errors induced by the theodolite intersection method and errors due to eccentricity of the observers are included in all of the data collected at sea. Statistics for all data sets are summarized in Appendix C

A. ABILITY OF THE OBSERVER

This factor is best evaluated by comparing the results of the data sets for each observer under similar conditions. The mean of the errors and the sample standard deviation are denoted by i and a respectively. Cruise I data is denoted by data sets beginning with IN in Table C-1 . The i for observer six seems extremely large. This was likely due to the seasickness that the individual experienced while measuring angles. As mentioned before, the sea conditions for Cruise I were extremely rough.

The data from Cruise II is shown in Table C-1 beginning with code IIS. The strong positive bias of the e's will be discussed in section F.

3^

The results of Zk data sets taken at Wharf Number 2 are indicated in Tables C-2 and C-3 by the codes beginning with WN and WS . These data sets show no major differences between the experienced and inexperienced observers.

Observer experience was also evaluated by comparing simultaneously observed sextant angles. This eliminated the effects of positioning error from the theodolite inter- sections. The mean of the differences for each group is shown below in Table IV for both cruises. Group 1 consisted of observers 1-3 and group 2 of observers k-6 . Two angle differences from group 1, Cruise I were rejected due to an obvious blunder in one angle of 10'. All angle differences involving observer 6, Cruise I were rejected due to their unreliability. The values in parentheses indicate the number of differences used to compute each mean. Some error was introduced due to the eccentricity of the three observers in each group. However, this error was assumed to be the same for each group.

Cruise I Cruise II

Group 1 2.5^4- (19) 1.29 (96) Group 2 3.10 (32) 1.66 (96)

TABLE IV. Mean Differences Between Simultaneously Measured Angles (Minutes of Arc).

35

The agreement of the group 1 observers was 0.3' "to 0.5' better than that of the group 2 observers. This was the best indication of improvement in measuring horizontal sextant angles due to increased experience.

B. DISTINCTNESS OF SIGNALS

The effect of signal clarity or distinctness was determined by comparing data collected without scopes (Cruise I) on the sextants to that collected with scopes (Cruise II). The data for G^ (angle 110/205) from Cruises I and II are denoted by IN- 0^ and IIS- 0^. The cumula- tive data from these data sets is shown graphically in Figure 5- This illustrates the increased dispersion of the Cruise I data compared to that of Cruise II.

•As a further comparison, inverse distances were determined between the main three-point sextant fixes and the corresponding check fixes. The results are shown below in Table V. Again the number of observations used in computing the statistics are shown in parentheses.

Cruise	I (10)	II (6^)
e	1.22	0.75
a	0.80	0.61

TABLE V. Statistics for Inverse Distances Between Main Three-Point Fixes and Check Fixes (Minutes of Arc) .

36

0.4

0.3

>- o

§0.2

LU 0£

iN-e,

NP=167 i=-0.32 a= 2.69

0.1

0,0 III' ymtr-^^/x

-10.0

0.4 r.

0.3

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

go. 2

LU

0.1

0.0

IIS-9, NP=256 £=+1.63 a= 1.58

-10.0

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

Figure 5- Data Distribution for 0i (Angle 110/205) - Cruise I (top) vs. Cruise II (bottom)

37

The decreased dispersion in the data from Cruise II versus that from Cruise I is due not only to the use of the scopes on the sextants, but also to the decrease in vessel motion. The data at Wharf Number 2 eliminated vessel motion completely. There were dramatic decreases in the values of e and a when scopes were added to the sextants. The cumulative statistics of the data are illustrated in Figure 6. Hence, the use of scopes on sextants decreases the magnitude of random errors in horizontal sextant angle measurement.

C. PLATFORM STABILITY

The effects of platform stability on random errors are best evaluated by comparing the Cruise I data to the data collected on Wharf Number 2 without scopes and comparing the Cruise II data to the data collected with scopes on Wharf Number 2 . These four sets of data are shown in Figures 5 and 6. The statistics are summarized below in Table VI.

	Cruise I	Wharf 2 (No Scopes)	Cruise II	Wharf 2 (Scopes)
i	-0.32	+0.89	+ 1.63	+ 0.20
0	2.69	2.07	1.58	0.6^

TABLE VI. Comparison of Cruise Data to Wharf Number 2 Data (Minutes of Arc).

38

0.4

wi

0.3

NP=360 £=+0.89 a= 2.07

0.4 r

0.3

0.2

-10.0

-5.0 0.0

ERROR (MINUTES)

10.0

0.1

0.0 I ' t ' I I I I I I I I I

ws

NP=360 G=+0.20 a= 0.64

rJ-f>^ i I I i I I l_J I I L_J L.

-5.0 0.0

ERROR (MINUTES)

5.0

10.0

Figure 6 . Distribution of Wharf Number 2- Data - Without Scopes (top) vs. With Scopes (bottom)

39

The increased dispersion of the cruise data is apparent in the figures as well as the table. Hence, when vessel stability is increased there is a significant reduction in the magnitude of random errors.

D. DIFFERENCES DUE TO DIRECTION OF ROTATION OF THE MICROMETER DRUM

The differences in the Wharf Number 2 data created by clockwise rotation of the micrometer drum versus counter- clockwise motion are shown in Tables C-2 and 0-3- Observations 1-15 were made with clockwise micrometer drum movement and correspond to decreasing values. Counter- clockwise rotation was used for observations I6-30. The magnitude of the differences in sample standard deviations is small - none are greater than 0.20' for the data taken with scopes and only 3 sore greater than 0.^0' for the no scope data. Observers 1 , 3 and 5 seem to measure smaller angles when using counterclockwise micrometer rotation. These three all have negative i values with average Ae's of -0.68', -0.60' and -0.50', respectively. The other observers did not exhibit this tendency as strongly.

The procedure for collecting index error information at the Beach Lab property was described on page 27 . The corresponding index correction (the negative of index errors) are summarized in Table C-A-. Scopes were used on the sextants. As before, observations I-I5 were made with clockwise micrometer drum movement and observations I6-30

^-0

with counterclockwise rotation. The magnitude of the differences in sample standard deviations is again small - all are less than O.I3' • Observers 1, 3 and 5 again measure slightly smaller angles when using counterclockwise micro- meter rotation. Their average Ai'sare -0.20', -O.32' and -0.48', respectively. In addition, observers 2 and k have negative Ae's. Therefore, the direction of rotation of the micrometer drum may introduce some small systematic error into sextant angle measurements.

E. INDEX ERROR DETERMINATION

Index errors were determined by each observer every day before measuring angles. The index corrections for this project are summarized in Table 0-5- Each index correction was derived using the method described by Umbach (1972). This procedure consists of holding the sextant vertically, observing the sea horizon, bringing the direct and reflected images into coincidence and reading the micrometer and vernier. This is repeated several times, alternately turning the micrometer drum clockwise and counterclockwise. The results are averaged, and this average is the index correction. From Table C-5 it is seen that index correc- tions for the same sextant vary between individuals by as much as two minutes. Even the same observer had differences of up to 1.7 minutes from day to day.

41

Instead of this method of daily determination of index corrections, the observations in Table C-4 can be averaged for each observer. The results of this are shown below in Table VII.

Observer 12 3^56

Sextant #2972 -O.50 -0.68 -0.86 -0.^6 -O.50 -0.74

Sextant #2982 -O.^il- -0.26 -O.^-k -0.25 -O.32 -0.53

Sextant #3003 -O.39 -O.38 -0.64 -O.09 -O.50 -0.64

TABLE VII. Index Corrections from Thirty Observations (Minutes of Arc) .

These results show differences from the index corrections determined by the daily method of as much as 1.5' indicating a systematic error. If there were no changes in index correction during the project the averages in Table VII should be more accurate than the daily determined correctors. Applying these newly determined index correctors would alter the sample means of all the previous data, but not the sample standard deviations. These observations suggest that each angleman should determine an index correction for his sextant by averaging ten to fifteen measurements in each direction thereby minimizing the magnitude of the systematic error.

42

F. CRUISE II SYSTEMATIC ERROR

Table C-1 showed that the observations of angle 110/205 during Cruise II had a positive bias of about 1.6'. Several possible error sources for this systematic error were considered and are described below:

(1) Mislocation of Signals 110 or 205: This did not occur since the angles between these stations during Cruise I showed no such positive bias.

(2) Consistent Errors in Theodolite Positioning: This could only occur if the theodolites were mispointed on the initial. This is highly unlikely due to the large number of such pointings and the good agreement with the azimuth check stations.

(3) Signals at Different Elevations: The excess angle measured due to differing elevations ranged from 2.5" to

3 '5" throughout the Cruise II work area. This was much smaller than the 1.6' bias.

(^) Incorrect Index Corrections: It is very unlikely that all six sextant observers could have made the same large mistake .

(5) Collimation Error: This error occurs when the scope is not parallel to the frame of the sextant and is always positive. No detectable collimation error was found when the sextants were examined using the method outlined in The Admiralty Manual of Hydrographic Surveying (I962).

43

(6) Incorrect Instrument Error: This value is determined for 15 degree increments of every sextant by the manufacturer and is attached to the inside of each sextant case. All three sextant cases had identical instru- ment errors posted for each angle - 0.0*. This suggested that the sextants should be checked for instrument errors.

The method suggested by CDR- Wintermyre and described earlier on page 27 was used in an attempt to evaluate the instrument error of all three sextants . Thirty observations were made with each sextant on two different angles. As determined by theodolite, angle one was 28051'22' and angle two was 47°03.00'. The differences between the theodolite measured angle and the sextant angle should only depend on theodolite error, index error and instrument error if the sextants are in otherwise good adjustment. The set of four theodolite measurements for each angle resulted in spreads of 03.6" and 07-8" respectively. Hence, theodolite errors were minimal. The mean index corrections that were deter- mined from 180 measurements for each sextant were -0.^0' for sextant #2972, -O.35' for #2982 and -0.45' for #3003. The index corrections arrived at by the author before using the sextant template were -0.66', -0.3^' and -0.5^' for the same three sextants. The difference between the theodolite angles and the mean of the JO sextant angles (corrected for the author's index corrections) are shown in Table VIII.

^4

Sextant Difference (Angle One) Difference (Angle Two) 2972 -0.72' -0.55'

2982 -0.65' -O.9O'

3003 -0.9I' -0.55'

TABLE VIII. Instrument Errors

It should be emphasized that the values in Table VIII are errors, not corrections. Therefore, the angles measured by the three sextants are 0.5' to 1.0' too small. This is contrary to the results from Cruise II which showed the sextant angles to be about 1.6' too large.

In summary, all the possible errors have been considered. The 1.6' bias results from some unknown source. Nevertheless, the standard deviation of the random errors from Cruise II is not affected by this error in the sample means.

^5

V. CONCLUSIONS

The standard deviation of the random errors in horizontal sextant measurements was found to vary mainly with the clarity of the signals being observed and the stability of the vessel. The differences due to observer experience were quite small for both the cruise data and the Wharf Number 2 data. Signal clarity was evaluated by measuring angles both with and without scopes on the sex- tants. This was isolated for only the Wharf Number 2 data. The cruise data showed the combined effects of signal clarity and platform stability and included some unknown amount of error due to theodolite positioning.

The general range of the sample standard deviations from the experimental data are summarized below in Table IX

Platform Stability Scopes/No Scopes Sample Standard

Deviation

Unstable (Cruise I) No Scopes 2. 3 '-3. 6'

Somewhat stable Scopes 1.3'-1.8' (Cruise II)

Very Stable No Scopes 0.9' -2.0' (Wharf Number 2)

Very Stable • Scopes 0.3' -0.8' (Wharf Number 2)

TABLE IX. Summary of Ranges of Sample Standard Deviations.

46

Although the above table illustrates the range of values for all observers, the cumulative statistics for all the data collected under each set of conditions give a better estimate of the magnitude of the random errors. The sample standard deviation for all angles measured with no scopes on an unstable platform is 2.69' and represents the probable upper bound. The lower limit for the sample — standard deviation was derived from the Wharf Number 2 data with scopes and is determined to be 0.6^'.

Index corrections for a given sextant may vary from individual to individual, but probably not by more than 0.5' • Some individuals measure consistently smaller or larger angles (by as much as 0.6') depending on the direc- tion of rotation of the micrometer drum. Therefore, for increased accuracy, index corrections should be determined by averaging ten or more measurements in each direction.

Some suggestions for further work in this area are appropriate. By using three T-2 theodolites to obtain a least square adjusted ship position, a more accurate set of statistics could be obtained for angles measured at sea. Various types of signals could be used at station locations to further determine the dependency of random error on signal clarity. The most variable quantity in attempting to quantify the random errors in horizontal sextant measure- ments is the ability of the observer. Although over 1^00 horizontal angles were analyzed, only six sextant observers

^7

were used. Further work with different observers would give a broader data base. Nevertheless, this study does present analytically derived values for the standard devia- tion of random errors in horizontal sextant measurements where only estimates existed before.

1^8

APPENDIX A: BLUNDERS AND SYSTEMATIC ERRORS AFFECTING THREE-POINT FIX POSITIONING ACCURACY

The potential blunders associated with three-point fixes include the following:

(1) Misread Sextant: This blunder is not readily identified for individual fixes. However, when conducting a hydrographic survey, consecutive fixes fall in a straight line if the vessel is carefully steered. A misread sextant angle will cause the fix to deviate from this line . The fix is then either rejected or an artificial position is created using dead reckoning.

(2) Misplotted Fix: This generally occurs only with manually plotted fixes and is identified by the same method as the error above. It is corrected by simply replotting the fix.

(3) Improper Identification of a Signal: This error, like the above two errors, is not easily discovered when only one fix is taken. Even along a carefully steered survey line, it may go undetected if the same erroneous signal is used throughout. But if the observer switches from that signal to a correct signal while on line the resulting fix will deviate from the straight line created by the previous fixes. The sextant data may be retained and the correct positions determined if the misidentif ied signal can be properly identified.

^9

The systematic errors that result in reduced accuracy of three-point fixes are as follows:

(1) Weak Fix Geometry: Strong geometry exists for a three-point fix when two of the three lines of position intersect at right angles. A fix has weak geometry when the three lines of positions approach coincidence. This greatly increases the effect of other errors on positional accuracy. Various fix geometries are discussed by Umbach (I976), Bowditch (1977) and Dedrick (1978). The effects

of weak fix geometry are minimized by following the general rule specified by these authors.

(2) Station Positions in Error: This error is similar to misidentifying a signal and its detection is also similar if the station position error is large. Small errors in station positions will often be undetected and will always be present in any three-point fix. Heinzen (1977) and Dedrick (1978) both discuss the three-point fix positional errors caused by incorrect station positions.

(3) Phase Error: The apparent displacement of a signal due to unequal illumination of its surface is called phase error. It is dependent on the shape of the signal, the angle of the sun with the line of sight, and the intensity of the sunlight. Water tanks may be especially susceptible to this kind of error. Formulas for correction of phase are usually not practicable due to the numerous factors upon which the correction depends [C-ossett 1971] •

50

(^) ' Observer and Observed Signals Not at the Same Elevation: The angle observed between signals with elevations differing from that of the observer are called inclined angles. This error is minimized by choosing signals that are at the same elevation as the observer. If this is not possible the inclined angle can be reduced to a horizontal angle by using the following formula from Umbach (1976):

cos C = CQs Q - sin(hi )sin(h?) cos (hi) cos (h2)

C = the horizontal angle

0 = the observed inclined angle

hi = the angular elevation of station 1 above the

observer h2 = the angular elevation of station 2 above the

observer

(5) Two-Observer Eccentric Error: This error is caused by the angle observers not being at exactly the same point. The magnitude is dependent on the distance between the observers and the angle of intersection of the two lines of position. Dedrick (I978) discusses this error and shows that for a separation distance of 3-0 feet and a 50° angle of intersection, the maximum error in the position is about ^.3 feet. It is minimized by selecting strong fix geometry and by having the angle observers stand as close together

as possible.

(6) Horizontal Refraction: Differences in the density of air along a line of sight can cause bending or refraction

51

of light rays. Vertical refraction is usually larger than horizontal refraction due to the air being stratified with denser layers near the ground. These layers are not hori- zontal over terrain that is sloping or unevenly heated and hence, horizontal refraction occurs. A line of sight pass- ing partly over water and partly over land is an example of unevenly heated terrain. Errors due to horizontal refrac- tion can be as large as 10 to 18 seconds of arc [Gossett

1971] .

(7) Sextant Parallax: This is caused by the separation between the center of the index mirror and the line joining the telescope axis and horizon glass (usually about ^ to 6 cm). It decreases as the range to the station increases. For a separation of ^.3 cm the parallax correction decreases from 0.^9' of arc at 1000 feet to 0.05' of arc at 10,000 feet [Dedrick 1978] .

(8) Sextant Errors: There are seven sources of error in the modern sounding sextant - ^ adjustable and 3 nonadjust- able [Bowditch 1977] • One nonadjustable error is called prismatic error and results from the two faces of the mirrors not being parallel. The other two nonadjustable errors are graduation errors (due to the arc or micrometer being improperly cut) and centering error (due to the index arm not pivoting at the exact center of curvature) . These are usually combined into one error called instrument error for which the manufacturer provides a correction table. The

52

adjustable errors are those resulting from nonperpendic- ularity of (1) the frame and the index mirror, and (2) the frame and horizon glass (side error) and the lack of parallelism between, (3) the index mirror and horizon glass at zero setting (index error), and (4) the telescope to the frame (collimation error). Bowditch (1977) explains each of these errors in detail and methods of adjustment to minimize them.

53

APPENDIX B: THEODOLITE INTERSECTION POSITION ERROR

The best estimate for each sextant angle observed aboard ACANIA was derived from the corresponding T-2 theodolite determined position. Errors in these positions caused inaccuracies in the best estimates of the angles. The magnitude of these positional errors was dependent on the angular resolution of the theodolite and the distance from the theodolite to the position. A well-trained observer using a T-2 theodolite on a stationary target during day- light hours can measure an angle within -2.5" ninety percent of the time when sixteen plate settings are used [Cervarich 1966]. This yields a standard deviation of 1.5"' However, for only one observation on a moving target such as ACANIA, a larger value must be used. Heinzen (1977) states that the angular error in measuring azimuths for hydrographic vessel positioning is -36 seconds. He does not present the method used to derive this value, what probability is associated with it or to what instrument it applies. Experience indicates that this value is quite large, but to derive the largest expected error in theodolite positioning of a moving target it was assumed that the standard deviation of the T-2 theodolite measurements was 36 seconds. A more reasonable value of 20 seconds was also used for comparison.

5^

The theodolite positioning error had varying effects on the computed best estimates used for the sextant angles. If displacement was along the circular line of position deter- mined by the ship and the two signals then the error in the best estimate was zero. Displacement normal to this line resulted in the maximum error. The maximum errors were determined for three points chosen near the extremes of the work area and are summarized in Tables B-1 and B-2 . The derivation of the values in these tables follows.

The root mean square error (d^ms) o^ 3- position is the square root of the sum of the squares of the standard deviations along the major and minor axes of a probability

ellipse. This is given by the equation 1 d-^i^s ~\l^t "^ ^J The values a^ and Oy are not the same as the standard deviations of the errors in the lines of position which are given by o^ and 02- However, for two independent lines of position, they are related by the following two equations from Bowditch (1977):

ol = ^- I o^ + 0^ + \ko^ + Q^) - 4sin26a^a2']

X 2sin^6 1 2 V 1 ^ 12

y 2sin^6 1 2 V 1 2 12

6 is the angle of cut or angle of intersection of the two lines of position. The result of substituting these equations into the formula for root mean square error is

sm 6 V 1 2

55

For an azimuthal line of position the standard deviation of the error is of the form a i = r j_ sin a for small values of a . rj_ is the range from the point to the theodolite station and a is the angular resolution or the standard deviation of the angular measurement. By substitution the root mean square error for azimuthal systems is finally

given by 1 drms = 5^^

sm

^f!

^ + r^ . The probability

2

associated with root mean square error is not constant but varies with the relationship between a^ and Oy. The probabilities in Table B-1 that result from the three values of Qy/ a were derived from Bowditch (1977) •

The errors in the best estimates of the angles ( 9i, 0 2 and 9 3) in Table B-2 were derived by contouring the errors which resulted from shifting the theodolite deter- mined positions + 1 meter in latitude and longitude. An

example is shown below in Figure B-1 .

-!• 0' +1' /.18'

^^1.53'

+ 1 .71 •

X

Figure B-1 . Angular Error at Position 1 Due to 2 d^ms

Theodolite Positioning Error of 0.59^ Meters

56

POSITION		1	2	3
Latitude (N) 36 Longitude (W) 121	O37'09.825" °53'30.088"	36036'^8.222" 121°53'03.28^"	36O37'07.210" 121°52'31.133"
ri		106^m	1933m	2520m
^2		1171m	715m	1701m
e		68.60	72.^0	^7.1°
ai		20"	20"	20"
^1		.103m	.187m	.2li,k-m
^2		.ll^m	.069m	.165m
^x		.137m	.198m	.37^m
^y		.092m	.069m	.1^7m
V^x		.674	.3^8	.393
^ ^rms		.165m	.210m	.^02m
Prob. (1	^rms^	6i^.3^	67.^?^	67. 2f.
2 ^rms		.330m	.^20m	.80^m
Prob. (2	^rms)	97. 6f.	96. 3f-	96. 5?^
a2		36"	36"	36"
^1		.186m	.337m	.^^Om
02		.20^1-111	.125m	.297m
^x		.2^6in	.356m	.675m
^y		.166m	.12i^^m	.26iJ'm
^y/^x		.67^	.3^9	.392
^ ^rms		.297m	.377m	.725m
Prob. (1	^^rms)	6^.3?^	67.^%	67.2^
2 ^rms		.59^m	.75^m	1.^50m
Prob. (2	^rms)	97. 6f.	96. 3f-	96.5f«
TABLE B-:	1 . Theo	dolite Posi'	tioning Errors	at Three

Locations

57

POSITION

Latitude (N) 36^37 '09.825" 36036*^8 .222" 36O37'07 .210" Longitude (W) 121°53'30 .088" 121O53'03 .28^^-" 121°52 '31 .I33"

aj		20"		20"		20"
2 ^rms	0	.330m	0	.^20m	0	.804m
Q-^ error		0.5'		1.9-		1.2'
©2 error		1.5'		2.1'		1.2'
0o error		1.7'		2.2'		1.3'
ag		36"		36"		36"
2 "^rms	0	.59km	0	.75^m	1	.450m
©1 error		0.9'		3.5'		2.1'
©2 error		2.7'		3.8'		2.1'
©o error		3.O'		3.9'		2.3'

TABLE B-2.

Maximum Errors in Angular Best Estimates at Three Locations Due to Theodolite Positioning Errors .

58

The origin in the above figure represents position 1. The grid around the origin are 1 meter shifts in latitude and longitude. The value at each grid point is the differ- ence in the angle 9^ at that point compared to the origin. By drawing the appropriate d27ms circle (in this case 2 d^ms = 0.59^ni) the corresponding angular error can be found.

It can be seen from Table B-2 that the angular errors induced by the theodolite positioning errors are not insignificant. However, the precise amount of error in the best estimates is indeterminant since the angular resolution of the T-2 theodolite angles is not known. The average distance between the theodolite determined positions and the corresponding mean sextant fixes for both cruises was 1.05m. In conclusion, the random errors in the horizontal sextant angles measured aboard ACANIA were made larger by the errors due to theodolite positioning. Hence, the values of standard deviation for the cruise data are the maximum expected errors for sextant angles measured at sea.

59

APPENDIX C: DATA SET STATISTICS

The coding system used for the data sets is shown below. All values shown are in minutes of arc.

(1) Data Origin:

I - Cruise I

II - Cruise II

W - Wharf Number 2 B - Beach Lab

(2) Use of Scopes on Sextants: N - no scopes

S - with scopes

(3) Observer Number (experience level decreases as

this number increases):

1 through 6 (^) Angle Designator:

Q I - 110/205

^2 - 205/202

03 - 205/107

©4 - 206/110

05 - 110/302 (5) Repeated Data Set (if necessary):

(1) - first set

(2) - second set

60

TOTAL INl-Gj 1^2-Qi IN3-9 1 IN^-® i INf-®! IN6-3i

(32)

e +0.09 -0.83 -1.16 +0.36 -0.^^ +6.37

a 2.25 2.63 2.62 3.33 2.34 3.59

TOTAL INl-0i(2) IN2-Q2 IN3-®-:! (10) ^

e +0.08 +0.88 +1.26

^ 3.8^ 2.55 3.44

TOTAL IISl-©! IIS2-9i IIS3-®! IIS^-Qi IIS5-®1 IIS6-©1 (32)

e +0.93 +1.31 +1.85 +1.78 +0.88 +2.18

a 1.28 1.76 1.25 1.61 1.67 1.28

TOTAL IISl-0i(2) IIS2-Q2 IIS3-®? llSk-'^^ IIS5-®l(2) IIS6-02 (32) ^ ^

E +2.if0 -0.2^ +1.^1-8 -0.^5 +1.74 -0.73

a 1.58 1.41 1.67 1.77 1.58 1.05

TABLE C-1 . Cruise I and Cruise II Data.

61

CLOCKWISE

MICROMETER (15) WNl-9Zj, WN2-eij. WN3-0Zj. WN^-Oi|, mS-Ql^, WN6-ei|

e +2.88 +0.86 -0.26 -3-38 +O.63 +0.08

a 1.25 0.92 1.12 1.15 0.47 0.80

COUNTERCLOCKWISE MICROMETER (I5)

e +2. 5k +1.88 -0.9^ -2.6^ +0.20 -1.26

a 0.96 0.78 1.38 0.64 1.17 0.82

TOTAL (30) i

A,

o

Ai

CLOCKWISE

MICROMETER (15) WNI-O5 WN2-05 WN3-05 WN^-O^ WN5-05 WN6-05

i +^,.^3 +1.79 +1.77 +2.35 -0.51 +0.92

a 0.85 1.10 1.10 0.96 1.12 0.7^1-

+2.71	+1.37	-0.60	-3.01	+0.42	-0.59
1.11	0.98	1.28	0.99	0.90	1.05
-0.34	+ 1.02	-0.68	+0.7^^	-0.43	-1.34

COUNTERCLOCKWISE MICROMETER (15)

e /\

a

+2.62 +3.04 +0.48 +2.04 -0.90 +2.78 1.15 0.71 0.69 0.91 0.48 1.06

TOTAL (30)

e +3.52 +2.42 +1.13 +2.19 -0.70 +1.85

a 1.35 1.11 1.12 0.93 0.87 1.31

Ae -1.81 +1.25 -1.29 -0.31 -0.40 +1.86

TABLE C-2. Wharf Number 2 Data - Without Scopes

62

CLOCKWISE

MICROMETER (15) WSl-Oi^ WS2-0Z| WS3-GZ| WS^-Oz^, ]{iS5-Ql^ WS6-0i|,

e +1.04 +0.58 -0.43 +0.10 +0.83 +0.44

d 0.25 0.23 0.33 0.51 0.51 0.48

COUNTERCLOCKWISE

MICROMETER (I5)

i +0.66 +0.41 -0.62 -I.I6 +0.14 -O.O3

a 0.27 0.21 0.32 0.44 0.31 0.57

TOTAL (30)

e +0.85 +0.50 -0.52 -0.53 +0.49 +0.21

a 0.32 0.23 0.33 0.79 0.55 0.57

Ae -0.38 -0.17 -0.19 -1.26 -0.69 -0.47

CLOCKWISE

MICROMETER (15) WSI-O5 WS2-O5 WS3-05 WS4-05 WS5-O5 WS6-0-

£ +0.62 -0.37 +0.33 +0.04 +0.10 +0.66

a 0.18 0.24 0.14 0.42 0.28 0.46

COUNTERCLOCKWISE MICROMETER (15)

a	+0.44 0.30	+0.03 0.32	+0.10 0.25	-0.25 0.23	-0.36 0.22	+ 1.34 0.45
TOTAL (30)
i a Ae	+0.53 0.26 -0.18	-0.17 0.35 +0.40	+0.21 0.23 -0.23	-0.10 0.36 -0.29	-0.13 0.34 -0.46	+ 1.00 0.57 + 0.67

TABLE C-3. Wharf Number 2 Data - With Scopes

63

CLOCKWISE SEXTANT #2972

MICROMETER (15) BSl(l) BS2(1) BS3(1) BS^J-Cl) BS5(1) BS6(1)

i -0.^1-1 -0.59 -0.77 -0.35 -0.28 -0.91

a 0.28 0.25 0.21 0.25 0.2-4- O.kk

COUNTERCLOCKWISE MICROMETER (15)

e a

-0.59 -0.77 -0.9^ -0.56 -0.71 -0.58 0.33 0.1^ O.lii- 0.3^ 0.30 0.^2

TOTAL (30)

£

a

CLOCKWISE SEXTANT #2982

MICROMETER (15) BS1(2) BS2(2) BS3(2) BS^(2) BS5(2) BS6(2)

-0.50	-0.68	-0.85	-0.i^6	-0.50	-0.75
0.32	0.21	0.20	0.31	0.3^	0.14-5
-0.18	-0.18	-0.17	-0.21	-0.^3	+0.33

e

A

a

-0.28 -0.0^1- -0.25 +0.09 -0.09 -0.67 0.36 0.23 0.28 0.36 0.28 0.^5

COUNTERCLOCKWISE MICROMETER (15)

e -0.59 -0.14-7 -0.63 -0.59 -0.55 -0.39

a 0.34 0.2i^ 0.16 0.35 0.25 0.34

TOTAL (30)

e a

-0.^3	-0.25	-o.kk	-0.25	-0.32	-0.53
0.38	0.32	0.30	0.49	0.35	0.i^2
-0.31	+0.01	-0.38	-0.68	-0.^^-6	+ 0.2i^

TABLE C-4. Index Correction Differences

6k

CLOCKWISE SEXTANT #3003

MICROMETER (15) BS1(3) BS2(3) BS3(3) BS'4-(3) BS5(3) BS6(3)

i -0.33 -0.19 -0.'4'3 +0.39 -0.23 -0.63

a 0.31 0.20 0.23 0.39 0.30 0.63

COUNTERCLOCKWISE MICROMETER (15)

e a	-0.^5 0.37	-0.57 0.22	-0.b5 0.22	-0.57 0.30	-0.78 O.ij-O	-0.6^ 0.50
TOTAL (30)
e a	-0.39 0.3^ -0.12	-0.38 0.28 -0.38	-0.6^' 0.31 -0.^1	-0.09 0.60 -0.96	-0.50 0.^5 -0.55	-0.63 0.58 -0.01

TABLE C-^. Index Correction Differences (continued).

65

OBSERVER 1 2 3 -tf 5 6

WITH SCOPES

Sextant

#2972 -0.5 +0.1 +1.0 -1.0

-0.2 -1.0

-0.^■ -0.6

-0.5

#2982 -0.6 -0.5 -0.5 -0.5

-0.2 -0.6

-0.3 -0.2 -1.0

#3003 -0.6 0.0 — +o.k' 0.0 +0.3

-0.5 -0.5

-0.5 -0.6

-0.-4- -0.5

-0.7 -0.5

NO SCOPES

Sextant

#2972 +0.5 -0.5 +0.5 -0.2 -1.0

-0.5 -1.2 -1.0 -O.ij-

#2982 : -O.l^ -1.0 -1.0 +0.6 +0.8

-1.0 -0.8 +1.0

-1.0

#3003 -0.5 +0.^1- -0.3 +0.2

-0.5 -1.0

-0.4 -0.5

-0.6 -0.6

-0.4

TABLE C-5. Abstract of Index Correctors From Cruise and Wharf Number 2 Data.

66

BIBLIOGRAPHY

Admiralty Manual of Hydrographic Surveying, v. 1, p. 297-307, Hydrographer of the Navy, London, I965.

Army Engineer Geodesy Intelligence and Mapping Research and Development Agency Report Number 3O-TR, Practical Field Accuracy Limits for a Wild T-2 Theodolite, by P.J. Cervarich II, August I966.

Blair, C., Coastal/Offshore Positioning by Optical Methods, paper presented at American Society of Civil Engineers Convention, Boston, Massachusetts, 2 April 1979-

Bodnar, A.N., NOAA - Pacific Marine Center, Seattle, private communications, December 1979 and June I98O.

Bowditch, N., American Practical Navigator, v. 1, Defense Mapping Agency Hydrographic Center, 1977'

Cotter, C.H., "A Brief History of the Method of Fixing by Horizontal Angles", Journal of Navigation, v. 25, no. k, p. 528-53^, October 1972.

Dedrick, K., Analysis of the Three-Point Fix, Report No. 77-3, Sacramento, June 1978.

Goodwin, E.M. and Kemp, J. F., "Accuracy Contours for Horizontal Angle Position Lines" , Journal of Navigation, V. 26, no. ^, p. 481-^85, October 1973-

Gossett, F.R., Manual of Geodetic Triangulation, Coast and Geodetic Survey Special Publication No. 2^7, p. 13^-136, U.S. Department of Commerce, 1971*

Heinzen, M.R., Hydrographic Surveys; Geodetic Control Criteria, Master's Thesis, Cornell University, Ithaca, New York, 1977-

Ingham, A.E., Hydrography for the Surveyor and Engineer, p. 21-25, John Wiley & Sons, 197^-

Ingham, A.E., Sea Surveying, v. 1 and 2, p. 71-90, John Wiley & Sons, 1975-

May, W.E., A History of Marine Navigation. 1st ed., p. 1^0, W.W. Norton & Company, 1973-

67

Mueller, I.I. and Ramsayer, K.H., Introduction to Surveying, p. 10-11, Frederick Ungar Publishing Co., 1979-

Umbach, M.J., Hydrographic Manual, ^th ed., U.S. Department of Commerce, National Oceanic and Atmospheric Adminis- tration, 1976.

University of New Brunswick-Department of Surveying Engineering-Lecture Note No. 45. Hydrographic Surveying I, by D.B. Thomson and D.E. Wells, p. 65, November 1977-

U.S. Coast Guard, COMDTINST M I65OO.I, Aids to Navigation Manual, Positioning, p. 1-C, I978.

U.S. Coast Guard Report No. CC-D-l06-7^. An Analysis of the Positioning Accuracy of Horizontal Sextant Angles, by J.T. Tozzi and H.E. Millan, June 197^-

68

INITIAL DISTRIBUTION LIST

No. Copies

1 . Defense Technical Information Center Cameron Station

Alexandria, VA 2231^1-

2. Library, Code 0142 Naval Postgraduate School Monterey, CA 939^0

3 • Chairman

Department of Oceanography - Code 68 Naval Postgraduate School Monterey, CA 939^0

k . Chairman

Department of Meteorology - Code 63 Naval Postgraduate School Monterey, CA 939^0

5. LCDR Dudley Leath

Naval Postgraduate School - Code 68Lh Monterey, CA 939^0

6. CDR Donald E. Nortrup Commanding Officer NOAA Ship Peirce

i^39 West York St. Norfolk, VA 235IO

7. LCDR Gerald B. Mills

Naval Postgraduate School - Code 68Mi Monterey, CA 939^0

8 . Director

Naval Oceanography Division Navy Observatory

34th and Massachusetts Avenue NW Washington, D.C 2039O

9- Commander

Naval Oceanography Command

NSTL Station

Bay St. Louis, MS 39529

69

10. Commanding Officer

Naval Oceanographic Office

NSTL Station

Bay St. Louis, MS 39529

11. Commanding Officer

Naval Ocean Research and Development

Activity NSTL Station Bay St. Louis, MS 39529

12. Director (Code PPH) Defense Mapping Agency

Bldg. 56, U.S. Naval Observatory Washington, D.C 20305

13. Director (Code HO)

Defense Mapping Agency Hydrographic

Topographic Center 6500 Brookes Lane Washington, D.C 203I5

1^. Director (Code PSD-MC) Defense Mapping School Ft. Belvoir, VA 22060

15- Director

National Ocean Survey (C) National Oceanic and Atmospheric

Administration Rockville, IVID 20852

16. Chief, Program Planning and Liaison

(NC-2) National Oceanic and Atmospheric

Administration Rockville, MD 20852

17. Chief, Marine Survey and Maps (C3) National Oceanic and Atmospheric

Administration Rockville, MD 20852

18. Director

Pacific Marine Center - NOAA 1801 Fairview Avenue East Seattle, WA 98102

19 • Director

Atlantic Marine Center - NOAA i^39 W. York Street Norfolk, VA 235IO

70

20. Commanding Officer Oceanographic Unit One USNS BOWDITCH (T-AGS21) Fleet Post Office

New York, NY 09501

21 . Commanding Officer Oceanographic Unit Two USNS BUTTON (T-AGS22) Fleet Post Office

San Francisco, CA 9660I

22. Commanding Officer Oceanographic Unit Three USNS H. H. HESS (T-AGS38) Fleet Post Office

San Francisco, CA 9660I

23. Commanding Officer Oceanographic Unit Four USNS CHAUVENET (T-AGS29) FPO, San Francisco, CA 966OI

21^'. CDR R. A. Anawalt

Chairman, Oceanography Department U.S. Naval Academy Annapolis, MD 21ift)2

25. LT Kenneth Perrin NOAA Ship Mt. Mitchell ^1-39 West York Street Norfolk, VA 235IO

26. LCDR Donald Winter 105 Moreell Circle Monterey, CA 939^0

27. Ms. Patricia Eaton

300 Glenwood Circle #133 Monterey, CA 939^0

28. LCDR Francisco Abreu Institute Hidrografico Rua Das Trinas , ^9 Lisbon - 2

Portugal

29. LTJG Luis Faria Institute Hydrografico Rua Das Trinas, ^9 Lisbon - 2

Portugal

71

30. LT Ali Kaplan Bostanci Kbyu Gonen - Balikesir Turkey

72

Th

esis

Thesis c.l

190^24

Mills

Analysis of random errors in horizontal sextant angles.

♦ UQ^ 9Z S^Vvix.

2 723^9 2723> ' 27235'

^

Thesis 190624

M5945 Mills

c.l Analysis of random

errors in horizontal

sextant angles.