Using FB to download Yahoo stock market data

General FreeBASIC programming questions.
Post Reply
DoomKlaxon
Posts: 5
Joined: May 12, 2012 0:20
Location: Duluth, Minnesota, USA

Using FB to download Yahoo stock market data

Post by DoomKlaxon »

Hi Everyone -

My first post on the forum. I have an Excel spreadsheet which I set up to download current stock market data from the Yahoo site for managing my 401(k). I want to convert this to a stand-alone program, and post it here so others can use it or port it to different systems.

Since I know essentially nothing about Internet programming, and the Excel VBA macro offers few clues, I searched the Web for examples and found the following method on a Russian site (whose url I have unfortunately misplaced). I replaced their example download url with a shortened version of my Yahoo one and it worked on the first try. Love when that happens. If I can find that Russian site again I’ll thank them. (I tried the HttpGet.bas example that came with FB but couldn’t get it to work on the Yahoo download although it worked on others).

I’ve sprinkled a few of my own comments here and there in the listing. Before I go any further with this program, I’m wondering if the Internet experts on this board would be willing look at it and suggest improvements. For my simple purpose, I think some of the code could be deleted. For one thing, the HttpQueryInfo function returns an error. Am I correct in guessing this is because the data is not actually in a file? When Yahoo gets my query they gather the info and zap it back to me as a stream of CSVs (Comma-Separated Values), and until that point no one can say what the length of the “file” is? Also, the listing mentions “IE 4.0 or better” but is IE even necessary? I certainly didn’t have to start up mine to make this work; just connected on my rural dial-up and ran the program. Or am I missing the point?

Any suggestions would be much appreciated. I’m running FB 0.21.1 and Windows XP. I downloaded a bunch of Wininet documentation off the MS website which I’ll try to wade through in quest of enlightenment. I only recently moved to FreeBasic, but I’ve been hobbying with various BASICs for many years. Just never did any web stuff.

Thanks -
Jim

Code: Select all

'download files, Windows only  (Russian note)
' requires IE 4.0 or better  (Russian note)
' tested using fb v.16b,  (Russian note)
'   and fb v.17b from CVS (11/2006)  (Russian note)

#include ONCE "windows.bi"
#INCLUDE ONCE "win\wininet.bi"

DECLARE FUNCTION dlfile(sURL AS STRING,LFileName AS STRING) AS INTEGER
DIM AS STRING sURL,LFileName
DIM AS INTEGER res

'''This is the download url for Yahoo's current/closing stock data in CSV format, for 3 stocks:
sURL = "http://download.finance.yahoo.com/d/quotes.csv?s=GE+IBM+PG&f=l1p2p5b4m3edn"
LFileName = EXEPATH & "\" & "YahDld02.txt"   'Location of downloaded data file on user's computer.

SCREENRES 640, 150, 24, 1
res = dlfile(sURL,LFileName)

IF res > 0 THEN
	LOCATE 1, 1
	PRINT LFileName; "  ...downloaded"
	PRINT " Size:"; res; " bytes"
ELSE
	'''I Get following error on stock download try - but the CSV data comes in OK... 
	PRINT "Error encountered, res not > 0..."; res
ENDIF

PRINT
PRINT " Sleeping...press a key to exit."
BEEP
SLEEP
END

FUNCTION dlfile(sURL AS STRING,LFileName AS STRING) AS INTEGER
	DIM AS HINTERNET hOpen, hFile
	DIM AS INTEGER ctot, nret, res, tbsize, t   'ctot is total number of bytes returned.
	                                            'nret is number of bytes returned (in each line?)???
	DIM AS STRING tbuff, ts
	DIM AS STRING scUserAgent = "Zippy"   '''Can this be any name I choose?
	DIM AS BYTE PTR mybuff
	mybuff = ALLOCATE(1024)

  'Create an internet connection.
  LOCATE 8, 26
  PRINT " Getting Internet Connection";
  hOpen = INTERNETOPEN(scUserAgent,internet_open_type_direct,NULL,NULL,0)
  IF hOpen = 0 THEN RETURN -1   'failed.

  'Open the url.
  LOCATE 8, 1
  PRINT SPACE(79);
  LOCATE 8, 35
  PRINT " Opening URL";
  hFile = INTERNETOPENURL(hOpen,sURL,NULL,0,internet_flag_reload,0)
  IF hFile = 0 THEN RETURN -2   'failed.

  'Let's get the file size.
  ' I think this requires IE 4.0 engine, not 3.0  (Russian note)
  tbuff = SPACE(32)
  tbsize = 32
  res = HTTPQUERYINFO(hFile,_
                    http_query_content_length,_
                    STRPTR(tbuff),_
                    @tbsize,_
                    NULL)
  
  tbsize=VAL(TRIM(tbuff))
  'if res = false or tbsize < 1 then return -3 'failed, but don't care..  (Russian note)
  
  IF tbsize > 0 THEN
  	t = 40 - (LEN(sURL)/2)
  	t = IIF(t < 1,1,t)
  	LOCATE 8, t
  	PRINT LEFT(sURL,80);
  	t = 40 - (LEN(LFileName)/2)
  	t = IIF(t < 1,1,t)
  	LOCATE 14, t
  	PRINT LFileName;
  	COLOR RGB(0,0,255), 0
  	LOCATE 10, 4
  	PRINT CHR(221);
  	LOCATE 10, 76
  	PRINT CHR(222);
  	COLOR RGB(0,255,0), 0
  ENDIF

  OPEN LFileName FOR BINARY ACCESS write AS #1
  nret = 99
  while nret > 0
  	res = INTERNETREADFILE(hFile,mybuff,1024,@nret)
  	IF nret > 0 THEN
  		PUT #1, , mybuff[0], nret
  		ctot += nret
  		IF tbsize > 0 THEN
  			LOCATE 10, 5 + INT(ctot/tbsize*70)
  			PRINT CHR(219);
  			ts = STR(ctot) & " bytes of " & STR(tbsize)
  			LOCATE 12, (40 - LEN(ts)/2)
  			PRINT ts;
  		 ELSE
  			LOCATE 1, 1
  			PRINT " Retrieved "; ctot; " bytes ";
  		ENDIF
  	ENDIF
  WEND
  CLOSE #1

  INTERNETCLOSEHANDLE(hFile)
  INTERNETCLOSEHANDLE(hOpen)
  DEALLOCATE mybuff
  RETURN tbsize
END FUNCTION
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

Hallo Jim, welcome at the forum.
DoomKlaxon wrote:Also, the listing mentions “IE 4.0 or better” but is IE even necessary?
AFAIK windows uses some functions of the IE for networking. That's why you cannot remove the IE from an XP installation. (But I'm no win expert.)

If you want to go without IE try another networking library. I recomment to check TSNE. It's based on the C runtime and works cross-platform. Find a client example here:
And here's the link to the neccessary TSNE_V3.bi file:
@Admin:

I think this thread should be moved to the General forum.
ThePuppetMaster
Posts: 22
Joined: Nov 04, 2008 21:20
Location: Germany
Contact:

Re: Using FB to download Yahoo stock market data

Post by ThePuppetMaster »

Hi. a english documented version of the test_client u can find here: http://www.freebasic-portal.de/porticul ... -1515.html


MfG
TPM
DoomKlaxon
Posts: 5
Joined: May 12, 2012 0:20
Location: Duluth, Minnesota, USA

Re: Using FB to download Yahoo stock market data

Post by DoomKlaxon »

Hey, PuppetMaster, thank you kindly.

I see it there; I will download it when I get home and try to take it somewhere. I am a total newbie to Internet programming, so this could get ugly....

I live near Duluth, Minnesota, USA. Where in Germany are you? I spent a college semester in Munich in 1970 (reveals age) but have long since forgotten the smattering of German I learned. (I was studying Russian, oddly enough.) Munich is a very nice town. I vividly recall taking an entire day once to stroll through the Deutsches Museum. And lots more....

- Jim
ThePuppetMaster
Posts: 22
Joined: Nov 04, 2008 21:20
Location: Germany
Contact:

Re: Using FB to download Yahoo stock market data

Post by ThePuppetMaster »

Hi @DoomKlaxon

well ... my english is poor too :P
I'm around 100 Km from munich away.

TSNE is special for "newbie's", so its perfect for u ;)

Its very easy to realize short and fast network sourcecodes. TSNE will make most of the stuff who a coder need for asynchronous network connections or a short app for downloading anywere. But, if u look around the example sources, i'm sure u see what i mean.
All Routines names by his function. After a short time u can use it intuitive.
Download a webpage is trouble free possible under 20 lines of source.

And, if u need help, simple ask. ;)


Greez
TPM

EDIT @TJF .. u post the wrong TSNE_V3 source link .. its: http://www.freebasic-portal.de/porticul ... -1459.html (see link-text: "...v3...")
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

ThePuppetMaster wrote: @TJF .. u post the wrong TSNE_V3 source link .. its: http://www.freebasic-portal.de/porticul ... -1459.html (see link-text: "...v3...")
Sorry, I allways struggle to find the link for the latest version. I think it's up to you to make your project page clearer (ie remove circled links). I recomment to create a new download page for TSNE where people can easily find the current version.

Anyway, I learned a lot from your code (other projects). Thanks for sharing.
ThePuppetMaster
Posts: 22
Joined: Nov 04, 2008 21:20
Location: Germany
Contact:

Re: Using FB to download Yahoo stock market data

Post by ThePuppetMaster »

np.

The latest Sourcecode version is always at the bottom of the Project-Page. http://www.freebasic-portal.de/projekte/tsne-11.html

sharing: no problem :) ... HF with it!


Greez
TPM
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

ThePuppetMaster wrote:The latest Sourcecode version is always at the bottom of the Project-Page.
You mean below the
Quellcode (Version: 2) ==VERALTET / (OLD Version) ==

Bitte nicht weiter nutzen! / Please don't use it futuremore!
lines?

I still think there is potential to optimize the project presentation.
ThePuppetMaster
Posts: 22
Joined: Nov 04, 2008 21:20
Location: Germany
Contact:

Re: Using FB to download Yahoo stock market data

Post by ThePuppetMaster »

Hi.

no... thats the "old" version.

well .. yeah .. the page structure of freebasic-portal for his projecs are a little bit crap.. its at the absolute bottom ... at the end of the page

looks like:
Dateimanager
FastPortscan.bas 19.03.09 03:35 7 kB [Update]
test_cookie.bas 28.06.09 17:52 12 kB [Update]
test_client_udp.bas 09.11.08 05:34 4 kB [Update]
test_server.bas 18.10.08 06:55 26 kB [Update]
test_client.bas 18.10.08 06:54 5 kB [Update]
test_smtp.bas 09.11.08 17:16 6 kB [Update]
-kein quellcode- 10.10.10 17:51 16 Bytes [Update]
test_server_bindipa.bas 23.02.11 23:57 27 kB [Update]
test_getWAN_IPA.bas 31.01.11 23:10 4 kB [Update]
api_doku.txt 02.10.11 06:16 31 kB [Update]
test_ping.bas 15.12.11 06:21 4 kB [Update]
TSNE_V3.bi 15.12.11 17:44 66 kB [Update]
test_client_en.bas 16.05.12 11:46 5 kB [Update]

EDIT ... i have now modified the Project page. Please tell me how its now.


Greez
TPM
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

ThePuppetMaster wrote:EDIT ... i have now modified the Project page. Please tell me how its now.
Much better!

Don you think the hint for fbc 0.18 is still important? What about prepending this hint to the source code and remove it from your front page?
ThePuppetMaster
Posts: 22
Joined: Nov 04, 2008 21:20
Location: Germany
Contact:

Re: Using FB to download Yahoo stock market data

Post by ThePuppetMaster »

thx :)

well ... maybe i remove it on 0.24 release :P


Greez
TPM
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

Code: Select all

#IF __FB_VERSION__ = "0.18.5" 
TYPE timeval
  tv_sec AS __time_t
  tv_usec AS __suseconds_t
END TYPE
#ELSE
#IFNDEF timeval
TYPE timeval
  tv_sec AS __time_t
  tv_usec AS __suseconds_t
END TYPE
#ENDIF
#ENDIF
:)
DoomKlaxon
Posts: 5
Joined: May 12, 2012 0:20
Location: Duluth, Minnesota, USA

Re: Using FB to download Yahoo stock market data

Post by DoomKlaxon »

Okay, I got my stock download thing working with TSNE v.3 over the weekend. I will post the code after I clean it up and add a bunch of comments about using the Yahoo site.
Thanks for your help, guys.
Jim
DoomKlaxon
Posts: 5
Joined: May 12, 2012 0:20
Location: Duluth, Minnesota, USA

Re: Using FB to download Yahoo stock market data

Post by DoomKlaxon »

Hello All -

Here’s code to get Yahoo! stock data using Martin Wiemann’s TSNE (TCP Socket Network Eventing) instead of Wininet.bi

I downloaded TSNE_V3 from
http://www.freebasic-portal.de/porticul ... -1459.html
and saved it as TSNE_V3.bi in my working directory. This program compiled and ran without error for me.

Code: Select all

'------------------------------------------------------------------------------------
'-  Download Yahoo! stock data using TSNE.  Jim Phillips  May 2012
'------------------------------------------------------------------------------------
#INCLUDE ONCE "TSNE_V3.bi"   'TCP Socket Network Eventing - Windows & Linux
                             'by Martin Wiemann
'------------------------------------------------------------------------------------
DIM G_Client AS UINTEGER                       'Create variable for the Client-Handle

'------------------------------------------------------------------------------------
'Callback routines
DECLARE SUB TSNE_Disconnected   (BYVAL V_TSNEID AS UINTEGER)
DECLARE SUB TSNE_Connected      (BYVAL V_TSNEID AS UINTEGER)
DECLARE SUB TSNE_NewData        (BYVAL V_TSNEID AS UINTEGER, _
                                 BYREF V_Data AS STRING)

'------------------------------------------------------------------------------------
DIM BV AS INTEGER                       'Variable for state return of function calls

'This creates a connection to the host site on port 80 (HTTP).
'The 60 value is for Timeout seconds, the recommended value for internet connections.
PRINT "[CONNECTING]"
BV = TSNE_Create_Client(G_Client, "download.finance.yahoo.com", 80, _
                        @TSNE_Disconnected, @TSNE_Connected, @TSNE_NEWDATA, 60)

'Check the return value
IF BV <> TSNE_Const_NoError THEN               'If an error,
    PRINT "[ERROR] " & TSNE_GetGURUCode(BV)    'print Human-Readable error message
    END -1                                     'and terminate program.
END IF
PRINT "[CONNECTION MADE]"                      'Else, it was ok.

PRINT "[WAIT]...."
TSNE_WaitClose(G_Client)         'Wait for the host to break the connection.

PRINT " * Press key to end. *"
SLEEP                            'So user can view screen printout.
PRINT "[END]"
END                              'Terminate program.

'------------------------------------------------------------------------------------
SUB TSNE_Disconnected(BYVAL V_TSNEID AS UINTEGER)
PRINT "[DISCONNECTED]"
END SUB

'------------------------------------------------------------------------------------
SUB TSNE_CONNECTED(BYVAL V_TSNEID AS UINTEGER)
'Build the data request string (for HTTP Protocol see Wikipedia).
'Also see notes() at end of listing.
DIM CRLF AS STRING = CHR(13, 10)
DIM D AS STRING
D += "GET /"                                        'Part of HTTP Get protocol.
D += "download.finance.yahoo.com/d/quotes.csv?s="   'Yahoo! download format. note(1)
D += "GE+Hon+IBM"                                   'Stock ticker symbols. note(2)
D += "&f="                                          'Yahoo! download format. note(1)
D += "snl1p2j1j6"                                   'Data fields requested. note(4)
D += " HTTP/1.0" & CRLF                             'Using HTTP 1.0 protocol.
D += "Host: download.finance.yahoo.com" & CRLF
D += "connection: close" & CRLF
D += CRLF                                           'HTTP Protocol needs blank line.
'Send the data request to Yahoo!.
PRINT "[SENDING REQUEST]...."
PRINT ">" & D & "<"
DIM BV AS INTEGER = TSNE_Data_Send(V_TSNEID, D)
IF BV <> TSNE_Const_NoError THEN
    PRINT "[ERROR]> " & TSNE_GetGURUCode(BV)
  ELSE
	  PRINT "[SEND OK]"
END IF
END SUB

'------------------------------------------------------------------------------------
SUB TSNE_NewData (BYVAL V_TSNEID AS UINTEGER, BYREF V_Data AS STRING)
PRINT "[DATA RECEIVED]>"
PRINT V_Data                           'string  V_Data  contains the stock data.
PRINT "<"
'Put the data in a file
OPEN "StockCSV.txt" FOR OUTPUT AS #1
PRINT #1, V_Data                       'note(3)
CLOSE #1
PRINT "Data Saved in File StockCSV.txt"
END SUB

'------------------------------------------------------------------------------------
'--------------- Notes regarding Yahoo! stock market data downloads -----------------
' (1) Yahoo! has changed the download URL occasionally in the past.  If you build a
'     program to use the site, I recommend including a way to change the host URL
'     and the format of the data request string.
' (2) In my and others' experience, the download can gag if you request more than
'     about 125 to 150 stocks at one time.  My watch list contains around 110 and
'     has not caused trouble.  If you want to download 1,000s of stocks, do 100 at a
'     time.  Don't download a pile of stocks 1 at a time, either; I have read this
'     will cause Yahoo! to drop your connection.  Yahoo! does not seem to care if
'     the ticker symbols are lower or upper case. 
' (3) Remember that the CSV file is saved as text; numeric values are strings.
'     Some fields, such as the company name, are always enclosed in quotation marks,
'     and may contain commas.  Sometimes a company's data is unavailable or its name
'     has changed for some reason, so don't assume fields will always be numeric or
'     string.  Normally numeric fields could become  ,"N/A",  or  ,N/A,  Try adding
'     a non-existent stock ticker and the e1 (error) code to see what kind of fields
'     you might deal with....  My example above will give you an inkling.
' --- The returned data is not 100% accurate; in my experience the error rate is
'     roughly 1%.  Before you decide to actually buy or sell a stock based on the
'     Yahoo! download, verify the numbers at another stock site.
' --- In my experience, price data can be up to 25+ minutes out of date.  But that's
'     true of other finance websites, too.
' --- There is a lot more on the Yahoo! site: historical prices, options, etc. that
'     I have not covered here.
' --- To my knowledge, this download site is not "officially" supported by Yahoo!.
' (4) It should be apparent how to build an arbitrary string of stock ticker symbols.
'     The data fields are another matter, and are determined by the following codes.
'     I don't claim to understand what every one of these is or does, and it seems
'     to me that some of them don't work - though I figure you'd have to be logged
'     into Yahoo! and looking at your Yahoo! portfolio for some of these to have any
'     meaning.  But here is every code I know of:
'     a   Ask                                 a2  Avg daily volume
'     a5  Ask Size                            b   Bid
'     b2  Ask Real-Time                       b3  Bid Real-Time
'     b4  Book Value                          b6  Bid Size
'     c   Change & Percent Change             c1  Change
'     c3  Commission                          c6  Change Real-Time
'     c8  Change After Hours Real-Time        d   Dividend Per Share
'     d1  Last Trade Date                     d2  Trade Date
'     e   Earnings Per Share                  e1  ERROR Indication
'     e7  EPS Estimate Current Year           e8  EPS Estimate Next Year
'     e9  EPS Estimate Next Quarter           f6  Float Shares
'     g   Day's Low                           g1  Holdings Gain Percent
'     g3  Annualized Gain                     g4  Holdings Gain
'     g5  Holdings Gain Percent Real-Time     g6  Holdings Gain Real-Time
'     h   Day's High                          i   More Info
'     i5  Order Book Real-Time                j   52-Week Low
'     j1  Market Capitalization               j3  Market Cap Real-Time
'     j4  EBITDA                              j5  Change From 52-Week Low
'     j6  Percent Change From 52-Week Low     k   52-Week High
'     k1  Last Trade Real-Time With Time      k2  Change Percent Real-Time
'     k3  Last Trade Size                     k4  Change From 52-Week High
'     k5  Percent Change From 52-Week High    l   Last Trade With Time
'     l1  Last Trade Price Only               l2  High Limit
'     l3  Low Limit                           m   Day's Range
'     m2  Day's Range Real-Time               m3  50-Day Moving Avg
'     m4  200-Day Moving Avg                  m5  Change From 200-Day Moving Avg
'     m6  % Change From 200-Day Moving Avg    m7  Change From 50-Day Moving Avg
'     m8  % Change From 50-Day Moving Avg     n   Name
'     n4  Notes                               o   Open
'     p   Previous Close                      p1  Price Paid
'     p2  Change In Percent                   p5  Price / Sales
'     p6  Price / Book                        q   Ex-Dividend Date
'     r   P/E Ratio                           r1  Dividend Pay Date
'     r2  P/E Ratio Real-Time                 r5  PEG Ratio
'     r6  Price/EPS Estimate Current Year     r7  Price/EPS Estimate Next Year
'     s   Symbol                              s1  Shares Owned
'     s7  Short Ratio                         t1  Last Trade Time
'     t6  Trade Links                         t7  Ticker Trend
'     t8  1 Yr Target Price                   v   Volume
'     v1  Holdings Value                      v7  Holdings Value Real-Time
'     w   52-Week Range                       w1  Day's Value Change
'     w4  Day's Value Change Real-Time        x   Stock Exchange
'     y   Dividend Yield
'                                      Enjoy!
Some comments in the code are mine, some are ThePuppetMaster’s. Errors, if any, are mine. :| I’ve added information at the end of the listing to cover a few things about the Yahoo! site.

Obviously, this is only the bare bones of a program; I need to add a CSV (Comma-Separated-Values) parser, and the routines that organize and rate the stocks in my watch list, display the info in a nice scrolling screen, etc. etc. etc. If someone has suggestions for any of that, I’d love to see your code before I re-invent the wheel!

Keep in mind that the downloaded data fields are not always in the expected format, so the CSV parser will need to be “intelligent”. I see there’s one at
http://www.freebasic.net/forum/viewtopi ... csv#p33205
so I’ll give that a look.

TSNE is supposed to work in Linux also, if someone wants to give it a try.

Thank You TJF and ThePuppetMaster for your assistance!

- Jim
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: Using FB to download Yahoo stock market data

Post by TJF »

DoomKlaxon wrote:Thank You TJF and ThePuppetMaster for your assistance!
You're welcome. Thanks for sharing your code (and thanks TPM for TSNE). I think your code is a good example for the Tips and Tricks forum.
Post Reply