Needs an explanation for Indexer

[duffee]

A number of methods (at, select_columns) refer to the indexer without ever explaining what it is and the concept behind it. I've got a vague idea from reading around Numpy/Pandas, but I can't guess at why my code doesn't always do what I expect. Probably hidden away by the _internal data structure. It'll help people coming at this for the first time.

It could explain why these two methods stringify the same, but they don't coerce the same

$h1 = double $df->at(6)
[ 1,344.5 1,309.3 1,295.5 1,292.4 1,258]

$h2 = double $df->columns( 'Height' )
[0 1 2 3 4 5]

for inspiration - https://pandas.pydata.org/docs/user_guide/indexing.html

[mohawk2]

Feel like investigating by reading the code, and then capturing findings in another doc update?

[duffee]

Unfortunately, I'll be starting a new job in a few weeks, so that investigation will get pushed onto the back burner (I've made a note for when I get around to writing up a blog post on Data::Frame). This drive-by was a side effect of trying to make an HTML table to Data::Frame converter, like Pandas has.

[mohawk2]

Fair enough. Congrats on the new job, and hope you'll be at LPW.

Can you point at some docs for the HTML scraper that Pandas has?

[duffee]

I'm aiming for LPW, but haven't thought about what I should prepare for it.

Using any library to download the html or given one saved on the filesystem, it uses pandas.read_html to extract the data. The note in McKinney's Python for Data Analysis says "by default it searches for and attempts to parse all tabular data contained within <table> tags. The result is a list of DataFrame objects" like so

tables = pd.read_html("file.html")
print tables[0].head()

A small headache is that my current script slurps everything into a Data::Frame as a PDL::SV, even the numeric data, meaning that you can't get a nice summary of the data as it thinks they're all strings.

Another example that fetches a page https://stackoverflow.com/questions/39120853/converting-an-html-table-in-pandas-dataframe

[mohawk2]

I'm aiming for LPW, but haven't thought about what I should prepare for it.

I'd hope it's fine to just turn up without preparing. I wasn't planning to prepare anything.

Using any library to download the html or given one saved on the filesystem, it uses pandas.read_html to extract the data. The note in McKinney's Python for Data Analysis says "by default it searches for and attempts to parse all tabular data contained within <table> tags. The result is a list of DataFrame objects" like so

tables = pd.read_html("file.html")
print tables[0].head()

A small headache is that my current script slurps everything into a Data::Frame as a PDL::SV, even the numeric data, meaning that you can't get a nice summary of the data as it thinks they're all strings.

Another example that fetches a page https://stackoverflow.com/questions/39120853/converting-an-html-table-in-pandas-dataframe

Cool! I'd encourage you to put what you've got somewhere (maybe on this issue?) to inspire others.