Bartz v. Anthropic — Slack exhibits

I was imagining just making a separate language and code dataset, and then training on a mix?

Yep that's the plan

But for the encoding itself it needed to be shared

Ah makes sense

My guess is that encodings aren't super sensitive to the precise mix, but I could be wrong, it's just a guess.

(So I'm not very worried)

I think that's right

My mental model is that performance is very insensitive to vocabulary size, and training on p% code and (1-p)% language is strictly better than training a p*n_vocab encoding on code and a (1-p)*n_vocab model on language, so the exact ratios will be really unimportant as long as they're within a factor of a few of 50/50.

https://www.reddit.com/r/LifeProTips/

https://www.reddit.com/r/NoStupidQuestions/

https://www.reddit.com/r/askscience/

https://www.reddit.com/r/BrandNewSentence/

Info on torrent structure and total sizes
2TB fiction
32TB nonfiction
68TB papers
https://wiki.archiveteam.org/index.php/Library_Genesis

attachment Screen Shot 2021-06-07 (335 KB), marked "Deleted on 06/07/2024 1:00 AM"

attachment Screen Shot 2021-06-07 (249 KB), marked "Deleted on 06/07/2024 1:00 AM"

metadata sample

[{'ID': '2331049',
'MD5': '641DC5DCE2FE9F2A12E3BBB9E9584C1E',
'Title': 'Sofia Petrovna',
'Author': 'Lydia Chukovskaya, Eliza Kellogg Klose (editor)',
'Series': '',
'Edition': 'Revised and Amended',
'Language': 'English',
'Year': '1988',
'Publisher': 'Northwestern University Press',
'Pages': '120',
'Identifier': '0810107945,9780810107946',
'GooglebookID': '',
'ASIN': '',
'Coverurl': '2331000/641dc5dce2fe9f2a12e3bbb9e9584c1e-g.jpg',
'Extension': 'pdf',
'Filesize': '19076046',
'Library': '',
'Issue': '',
'Locator': 'Sofia Petrovna _ Lydia Chukovsk - Chukovskaia, Lidiia Korneevna;W.pdf',
'Commentary': '',
'Generic': '',
'Visible': '',
'TimeAdded': '2021-06-06 02:20:32',
'TimeLastModified': '2021-06-07 03:00:03'}]

MD5 corresponds to the hash in the torrent

Identifier is the ISBN

fiction metadata dump:
REDACTEDdata/raw/libgen/fiction/metadata.csv.gz
REDACTEDdata/raw/libgen/nonfiction/metadata.csv.gz

Analysis notebook
Top file extensions for Fiction

[Row(Extension='epub', count=1296036),
Row(Extension='pdf', count=266084),
Row(Extension='mobi', count=237827),
Row(Extension='rar', count=117357),
Row(Extension='lit', count=64659),
Row(Extension='txt', count=57003),
Row(Extension='rtf', count=56605),
Row(Extension='doc', count=51297),
Row(Extension='azw3', count=49166),
Row(Extension='html', count=24270),
Row(Extension='zip', count=21490),
Row(Extension='htm', count=20965),
Row(Extension='fb2', count=20646),
Row(Extension='pdb', count=11734),
Row(Extension='lrf', count=10434),
Row(Extension='prc', count=4977),
Row(Extension='cbz', count=3996),
Row(Extension='htmlz', count=3904),
Row(Extension='docx', count=3095),
Row(Extension='cbr', count=2411),
Row(Extension='azw', count=1598),
Row(Extension='xml', count=870),
Row(Extension='odt', count=865),
Row(Extension='djvu', count=713),
Row(Extension='oeb', count=490),
Row(Extension='gz', count=453),
Row(Extension='txtz', count=395),

809306 with an identifier, which I think is the ISBN

top 10 extensions with identifiers

[Row(Extension='epub', count=571475),
Row(Extension='mobi', count=100185),
Row(Extension='pdf', count=64569),
Row(Extension='azw3', count=19300),
Row(Extension='fb2', count=13440),
Row(Extension='rtf', count=9119),
Row(Extension='lit', count=8324),
Row(Extension='txt', count=6802),
Row(Extension='rar', count=5068),
Row(Extension='lrf', count=3409)]

i'll start extracting these as "known high quality"

language-combined-evals description
"Raw evaluation data we're using as a quick proxy for performance, combining "
"hellaswag, triviaqa, squad, winogrande, lambada, openbookqa, and arc, REDACTED
REDACTED

Maybe it's a formatting issue with wikipedia, or maybe wikipedia is very helpful mixed in, but not great to train on by itself because of sufficiently different formatting or something

I was also surprised that books3 had much better transfer to python than webtext or wikipedia

attachment Screen Shot 2021-06-08 (284 KB), marked "Deleted on 06/08/2024 1:17 AM"

More excited for libgen after seeing how well books3 is doing in some ways.
I think it'd be helpful to split fiction and non-fiction, so I can study the quality/transfer of those two datasets seperately. They feel quite different.

Another view on wikipedia and webtext not doing great

attachment Screen Shot 2021-06-08 (245 KB), marked "Deleted on 06/08/2024 1:17 AM"

Next experiments I plan on doing are

1. A few mixtures. I wonder how much of the benefit for pile and canonical-8k are coming from having at least some small fraction of openwebtext2, books3, and enwikipedia.
2. Size scan.

@jared @ben

Super interesting!! Thanks for looking at this.

I don't have a strong prior belief here. I have a personal preference for books so I'm amused that they're doing well. The pile has a lot of academic papers and stuff in it as well. You can look up the paper to see the exact mixture.

Yeah, I was surprised by how academic the text in the pile is.

attachment Screen Shot 2021-06-08 (114 KB), marked "Deleted on 06/08/2024 1:17 AM"

It looks like canonical-8k is REDACTED books, REDACTED webtextREDACTED wikipedia, REDACTED pile. So makes sense why pile and canonical are so similar.

REDACTED
# Make sure we only get 1 copy of each of these.
["khan-math", 1e-8],
["hendrycks-math", 1e-8],
]
PILE = [
["mix", 100],
# Make sure we only get 1 copy of each of these.
["khan-math", 1e-8],
["hendrycks-math", 1e-8],
] https://github.com/anthropics/REDACTED

Are there any stats on what's in books3? Very curious why it's doing better on python. Could be worth just looking at 1000 shuffled titles and getting a rough feel. Maybe it's just line wrapping and indentation matching 😅

I haven't found stats, but from what I can tell books3 is all of bibliotik which seems like a private torrent network around booklike things. in plain text form. I think it includes magazines, textbooks, and niche text people on the internet would want...

People on the internet like programming textbooks, so that's my guess at what makes books3 good at programming.

https://github.com/soskek/bookcorpus/issues/27#issuecomment-716104208

It seems like the-eye is a group that works with the pile

https://the-eye.eu/public/Books/

I think these directories give you some idea of the kind of distribution books3 is, but don't know anything about the relative size.

There are some big top level categories like:

Poetry, SciFi_fantasy, History, Manuals, Medical

But there's also some much more specific top level categories that
4chan_pol_Archives, Asimov, Playboy, Radical Feminist Literature

Yea I think just glancing at the titles will be more informative for that reason. I can get you the bash command after I land in sf

It's interesting that the curves cross here, rather than maintaining the same rankings throughout

Yeah, the crossings of various curves makes me less sure what to expect when scaled up to larger models.

From their FAQ:

Q: Should the Z-Library collection be added to Library Genesis?
A: Yes! However, it is tricky. Library Genesis splits out its collection between non-fiction and fiction. They also have relatively high quality standards. If you are interested in organizing all the books to meet their requirements, let us know.

We will release the data in stages, as we are still processing the files. Right now the metadata file and a few of the torrents are available. Note that the torrent files are only available through our TOR mirror.

Presently, only the first four of several dozens of parts are available.

💥
Since Library Genesis is already preserved with public torrents, and is included in the Z-Library, we did a basic deduplication against Library Genesis in June 2022.

After the deduplication we are left with over 2 million files

Most files are legitimate ebooks or scans of books, but there might some malicious files in there. We didn't do a virus scan of the collection. It should be safe to download the torrents, but if you want to open any of the books, be sure to have updated your viewing software to the latest version, and don't run any executables.

zlibrary my beloved

@nova (not technically a bot) wanna help us scrape this? i already have a transmission automated setup ready to roll
https://github.com/anthropics/anthropicREDACTEDbitbox/scrape/libgen/libgen_babysitter.py

yes, are you doing this now or in a few hours? I'm about to head out

lol even in the next week would be great

take your weekend!

cool, will do

update on this, I have the torrent files and I'm kicking this off, haven't decided whether I'm going to try on gcp instead of AWS yet

Just make sure to set your upload rate to 0!

cc @Sheer El Showk

@nova (not technically a bot) this might be another motivation to set up an isolated cluster? happy to pair on that (maybe next week?).

excited about getting this data – seems like it might get us to our 8T target?

a lot of these might be scans which would reduce the overall token count and also require us to set up a tesseract (or alternative pipeline). don't mean to curb enthusiasm too much but just something to bear in mind.

These are currently downloading, ebs volume is vol-0e8d35e795abd072c, I expect to have most in ebs in the morning

Naive estimate of words downloaded based on a 1% sample extraction (index is the output of find):

nova-public-0 (unsynced) /mnt/zlib/pilimi$ du -sh .
987G .
nova-public-0 (unsynced) /mnt/zlib/pilimi$ wc -l index
496447 index
nova-public-0 (unsynced) /mnt/zlib/pilimi$ shuf index | head -n 5000 | xargs -I {} -P 20 sh -c 'pandoc {} -t markdown 2> /dev/null | wc -w || true' | perl -ne 'BEGIN{$c = 0;$i = 0;}$i++;chomp;$c+=$_;print $i."/5000\t".sprintf("%02d", $c/$i*`wc -l index`/1000000)."M\n"'
[SNIP]
4997/5000 2359M

Not sure why many of them are returning 0 - probably due to UTF errors which I was seeing. I'm running another extract using unoconv but I'm not putting much effort into it. Format is .fb.zip files, should be pandoccable with a bit of effort.

unoconv is buggy here for reasons that are unclear to me and I'm not that interested in re-estimating, @Sheer El Showk any preferences for conversions etc before I upload to S3?

i'd just leave it as fbzip and do the processing in spark

uploading at s3:/REDACTEDpilimi/

I was gonna start looking at these files but I'm worried about the possibility of malicious files. what measures do we want to take against this? is this the day we figure out a throw-away GCP cluster?

related to this: do we want to mirror some choice non-confidential buckets to gcp (such as data/raw) or do we just want to copy data over ad-hoc?

@eli any chance that you can terraform something temporary?

we can absolutely terraform a throw-away cluster

Ok, so step 1 is getting a token estimate

wc -l on the index gives ~500k files

Next trying to understand this command from Nova:

shuf index | head -n 5000 | xargs -I {} -P 20 sh -c 'pandoc {} -t markdown 2> /dev/null | wc -w || true' | perl -ne 'BEGIN{$c = 0;$i = 0;}$i++;chomp;$c+=$_;print $i."/5000\t".sprintf("%02d", $c/$i*`wc -l index`/1000000)."M\n"'It looks like this:

1. shuf index | head -n 5000 -> pick out 5000 random files
2. xargs -I {} -P 20 sh -c 'pandoc {} -t markdown 2> /dev/null | wc -w || true' - Run pandoc with 20 processes writing to markdown, counting words. Not sure what || true does, maybe only passes ones that have non-error output on?
3. Send these counts to a perl script which prints out progress and maybe counts words? A bit confused here.

Claude was helpful here, this seems to match up with Nova's claim that there were only 2.4B words. This is just in the current subset though, so it seems likely that even if we didn't do anything special we could get 240B words from this, which is larger than our libgen dataset. So that's exciting.

Next step is to address the pandoc issues though, we should be able to get much more data here

Claude downloaded 20 files for me (so useful when I don't know basics of a language like bash!). Main error now is:

pandoc 132906 -t markdown
pandoc: Cannot decode byte '\x87': Data.Text.Internal.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

Different files fail on different bytes

Ok, so I looked at the mime-types and got

binary
binary
binary
binary
binary
us-ascii
unknown-8bit
utf-8
utf-8
binary
utf-8
utf-8
application/mswordbinary
utf-8
binary
binary
binary
binary
binary
binary
binary
binary
us-ascii

binary utf-8 andus-ascii seem to be pandoc-able, whereas binary isn't. Figuring out what "binary" means seems like the main goal here.

Ok, according to pilimi faq, we should look at the mysql metadata table to figure out formats and such.

Seeing this if I cat the mysql file. I should probably load this into a MYSQL GUI of some sort, but I think I need security approval for that:

CREATE TABLE `books` (
`zlibrary_id` int NOT NULL,
`date_added` text NOT NULL,
`date_modified` text NOT NULL,
`extension` text NOT NULL,
`filesize` int DEFAULT NULL,
`filesize_reported` int NOT NULL,
`md5` char(32) DEFAULT NULL,
`md5_reported` char(32) NOT NULL,
`title` text NOT NULL,
`author` text NOT NULL,
`publisher` text NOT NULL,
`language` text NOT NULL,
`series` text NOT NULL,
`volume` text NOT NULL,
`edition` text NOT NULL,
`year` text NOT NULL,
`pages` text NOT NULL,
`description` text NOT NULL,
`cover_url` text NOT NULL,
PRIMARY KEY (`zlibrary_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
/*!40101 SET character_set_client = @saved_cs_client */;

Ok, some of them are fb2.zip files and unzip seems to work on them, still can't pandoc the unzipped file though

Ok, this took me a while, but updating pandoc seems to have enabled me to get these

Lots of this is non-english. Next step is to get the MYSQL table loaded up and see how much data is left after we filter for english, and likely filesize.

@nick from what I can tell pandoc can't handle pdfs so we're mostly targetting epubs right?

i've gotta go afk but I should be working again in 20-30 min if you want to just pair up on this

Sure!

I'm increasingly wishing we could have a few simple throw away aws dev boxes completely outside our cluster for just doing stuff like this. using docker it's trivial to standup a throwaway mysql server. alternatively maybe we could just run vmware on our laptops? @nova (not technically a bot) @ben?

turns out I'm in a very noisy place (dropped kids off so they can play)…happy to see how magical my headphones are but i also don't want to disrupt your work so I can also do something else and pair later if you prefer

Give me 2 minutes and then let's try pairing

for the command: in part 2, the true prevents it from quitting on a nonzero exit code from pandoc, part 3 is sending the list of word counts to a perl script that estimates how many words there gradually by scaling by the number of lines received and the proportion of the index

I'm up and can pair if that's helpful, I think this isn't much data so doing the processing at GCP would be sufficient safety

I think we are good for now! We are currently going through the mysql database of to try to figure out how much of each filetype / language, etc there are.

I assumed that's fine to do on a devbox since the mysql file is just metadata / made by pilimi, although we could switch to GCP for this if needed

I think it's fine for now, feel free to spin this sort of thing up in a sandbox though

Ok, the main result of pairing here is that it seems like:

1. PiLiMi has ~70B tokens of English, 90% of which is PDFs. It's unclear if these require OCR
2. There are about 120B tokens if we add in all the other languages

Overall seems less exciting than hoped for, but still likely worth getting.

did you convert the database to SQLite for easier future use?

also we can straightforwardly 7x this by waiting for the other drops

We didn't convert to SQLite, I'm not sure how to do that

Oh, this is only 1/7th of the data they have?

I still have the mysql database open on a devbox though

I think their mysql database is for the full dump. It seems to have ~2 million PDFs and 1.4 million epubs. The blog post says
After the deduplication we are left with over 2 million files, totalling just under 7TB.
The collection consists of two parts: a MySQL ".sql.gz" dump of the metadata, and the 72 torrent files of around 50-100GB each.

https://github.com/dumblob/mysql2sqlite

IIRC they haven't released all the torrent files yet

we can just dump the table to a csv and then import the csv into sqllite…happy to do this later (i'm afk right now)

That would be fine, it would just be nice to not have to pull up mysql for every little query

csv lets you use pandas which I think is probably easier for most ppl anyway

Reasonable, I'm just a SQL enjoyer 😅

https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html#pandas.read_sql
might be useful for pandas enjoyers then 😎

I've put the csv here and also created a small notion doc to help people setup a quick ad-hoc mysql server ( @nova (not technically a bot), @ben do we have security concerns around encouraging this?). happy for suggestions as to better homes for the dump and doc. cc @nick for notion doc

defer to nova on this one

@Sheer El Showk your notion doc would make more sense as a colab notebook?

no need to refactor this one, but for future reference

for reference i have a similar mysql flavored thing here
https://github.com/anthropics/anthropic/REDACTED

I think there should be a metadata.json file which has a dictionary mapping each file to n_examples. You can sum the values there.

I got this notification from a recent job: https://anthropic.slack.com/archives/REDACTED

looks like you also got a few recently https://anthropic.slack.com/archives/REDACTED

thanks @sam-pling…i saw the complete part but didn't realize there was the rest of that info. very cool!

@nick the sum is already in the metadata.json

spreadsheet version is here

just to clarify we already have 70b toks from the multilingual books. the raw datasets are in and I'm waiting on some spark capacity to finish the encoded datastes. Also for CC we should distinguish PDFs from websites. I think there's a fair amount of non-en PDFs we can easily pull out just by turning off the filter. I'll update here when I find the estimate but IIRC it's about the same as we had en PDFs (from CC).

here's the info on the additional CC pdfs we could get (both en and non-en) :
https://anthropic.slack.com/archives/REDACTED

Excellent! 🙌 How high quality do you think the CC PDFs are in general?

Not really sure. I think it's a mixed bag but I added a lot of heuristics to reduce crappy docs. I filter docs with less than 10 pages per doc and less than some num words per page. Without those filters they'd be way more docs

I wonder if there are any particularly good narrow filters of common crawl we can do?

e.g.:

1. make a list of news websites and filter for that
2. filter for pages that have mathjax or latex in it

Ben suggested just having a domain whitelist and fully crawling those

crawling from CC? or crawling ourselves?

(I think possible we've been a bit too reluctant to crawl things ourselves, not sure though)

Hm, I think getting it from CC is easier since it's already there and we just need to add some sort of rule like "if domain contains www.cnn.com, don't filter it at all"?

Unless we think CC is incomplete or messing something up, in which case crawling it ourselves would be useful

do we know how complete CC is? I have no idea

I have no idea either. We could crawl ourselves and crawl from CC for a couple example domains to find out.

@ben, do you know how complete common crawl is? also cc @nova (not technically a bot) @Joshua Landau since you have experience with this sort of thing

https://groups.google.com/g/common-crawl/c/xmSZX85cRjg/m/RYrdBn2EBAAJ

a monthly crawl now runs for 10-12 days on 100 EC2 instances.ooooh, good to know

My read on that thread is that it isn't very complete, which makes running our own crawls seem more useful

also sounds like archive.org might be significantly more complete?

Also doing a crawl on warcs should get us lots of extra pages if we want

could you elaborate on that? I thought we were already using WARCs?

Crawl on Wayback Machine WARCs

So all past versions of sites like CNN

trying to figure out where to get those WARCs now

https://github.com/hartator/wayback-machine-downloader I use this

also found this https://archive.org/details/@webcrawl

I'd be happy to pair on this if helpful (free for the next hour). If not, I might look around some more on internet archive to see if there are any other datasets that look large / promising.

it looks to me like you can directly see (e.g.) their crawls for each day, though it can't be directly downloaded

attachment image.png (811 KB; preview blank in export)

You can download those, yes

Just have to ask IA directly

@nick for news we can whitelist a bunch of news site domains to ensure we don't accidentally lose them from quality filter

i don't expect C4 to be very different from what we've got

I think the concern is that commoncrawl might not have all the news articles

you can check the columnar index

Looks like CC has a specific news subdirectory with daily news articles. Have we downloaded that?
https://commoncrawl.org/2016/10/news-dataset-available/

Don't think so! Good idea

titles.txt

attachment titles.txt (.txt | 116 KB)

any rough token estimates?

Nope, sorry

now might be the time to get data-atlas working again!

a five second notion search doesn't turn this up, point me somewhere? :blobthx:

here's the kind of thing it creates: https://s3-frontend.he.anthropic.comREDACTED

it's just a umap of a dataset subsample, using a tiny model, with mouseover to show the samples

I find that it's an easier way to qualitatively understand the contents of different datasets

search for https://s3-frontend.he.anthropic.com/REDACTED

(name comes from chris's " activation atlases", which is basically what this is)

one way we could use it for CC filtering is to color points by the filter score

I think
https://github.com/anthropics/anthropic/REDACTED

main thing that'd keep it from working right now is installing cuML, which is super annoying for some reason

Ok, that's unfortunate but still a pretty great dataset to have! Do we know why they have only been able to download 2m books? (I couldn't see any mention of that in the blog post) Do we think they will eventually get all of them?

Ah ok, it looks like its the dedup against libgen that is the problem:
Since Library Genesis is already preserved with public torrents, and is included in the Z-Library, we did a basic deduplication against Library Genesis in June 2022. For this we used MD5 hashes. There is likely a lot more duplicate content in the library, such as multiple file formats with the same book. This is hard to detect accurately, so we don't. After the deduplication we are left with over 2 million files, totalling just under 7TB. https://www.reddit.com/r/libgen/comments/vr5cq3/introducing_the_pirate_library_mirror_preserving/

I don't think it's the only problem … it looks like they can't access most of Z-library

well tbh this is not maximally clear
The first library that we have mirrored is Z-Library. This is a popular (and illegal) library. They have taken the Library Genesis collection and made it easily searchable. On top of that, they have become very effective at solliciting new book contributions, by incentivizing contributing users with various perks. They currently do not contribute these new books back to Library Genesis. And unlike Library Genesis, they do not make their collection easily mirrorable, which prevents wide preservation. This is important to their business model, since they charge money for accessing their collection in bulk (more than 10 books per day).I don't understand if this means they have managed to download most of it or not

Oh, I think this was just providing their motivations for mirroring it

I'd have to go back and check but I don't recall libgen having ~8m books which is what would be implied if they went from 10m to 2m by dedup. @ben had a thread/notebook on this but I'd have to find it.

but in practice I don't think anything more will be available in a finch timescale

sadly

I'm adding in more file-format which might give us a bit of a boost (we can handle anything calibre can and that's quite large)

Yep, would still be useful to find more for future models if it turns up, but that makes sense.

sorry for the false hope …i should have read through the pilimi site a bit before jumping the gun on their metadata

What's the updated token estimate / ETA for these? Like 70B english and maybe 50B other languages?

No worries, getting early estimates is super useful even if things turn out differently sometimes.

i'm doing some small tests and adding more formats right now. once the job finishes I will launch a large english job. might even launch en and non-en in parallel. note sure how long they take but I'm guessing done by tomorrow.

I am hopefully it will be more token than the above but its hard to say now. I can extrapolate again from doc count (assuming uniformity). I'll get to you with that in a sec.

ignore file formats (since we seem to support the most prevalent ones) we have ~700k en books and ~1.3m non-en books. comparing to @ben's ligben stats it seems we have 50% as much english but perhaps the same amount of non-en:

[Row(trim(Language)='English', count=1537983),
Row(trim(Language)='Russian', count=906758),
Row(trim(Language)='German', count=155041),
Row(trim(Language)='Spanish', count=64666),
Row(trim(Language)='French', count=39244),
Row(trim(Language)='Italian', count=37322),
Row(trim(Language)='Ukrainian', count=36302),
REDACTED

Sorry I'm having a bit of trouble tracing through what the 50% is relative to since there are a lot of numbers in that thread. I think we have 200B tokens of english libgen and 100B tokens of non-en (is that correct?), so you're saying this is maybe 100B tokens each of english and non-en?

sorry let me clarify. from my count we have ~700k english books in pilimi. this is ~50% of the number in libgen (1.5m from ben's doc above). so all else being equal we should expect 50% as many english tokens from pilimi.

but looking at non-english stuff we have 1.3m non-english books in pilimi which looks roughly comparable to the amount of non-en stuff in libgen so we should expect roughly the same number of non-english tokens.

Ok, makes sense!

it's already on s3 https://arxiv.org/help/bulk_data_s3

you'll need the requester pays flag https://stackoverflow.com/a/33019940

s5cmd --request-payer=requester ls s3://arxiv/src/

aws s3 ls --request-payer requester s3://arxiv/src/ worked for me, I don't think s5cmd has that option

i just used it locally

maybe i have a diff version

2742316622777 bytes total -> 2.5T

lots of that is probably images

> s5cmd version
v2.0.0-35bb2fa

brew install peak/tap/s5cmd

@Sheer El Showk

Sheer is going to take this on. Just to clarify the plan here:

1. Download all the LaTeX source files
2. Run Minerva preprocessing
   1. Concatenate multiple files
   2. remove anything before first section header or after appendix/bibliography header
   3. Add title and abstract from the ArXiV metadata
3. Run Minerva filtering on them, removing
   1. Longer than 75k tokens
   2. More than 0.6 tokens per character on average (I'm a bit confused what this means, isn't 1 token per char the min?)
   3. docs with no \section headers
   4. empty after preprocessing

Then we will just include 1 or 2 epochs of this in addition to 1 epoch of our current arxiv data.

@nick are we good using "words" as a proxy for tokens above? when we filter in dataset creation we don't normally tokenize (though I guess we can).

Yeah that seems fine. I think 1.5 * words might be better

@ben OK with me upgrading s5cmd directly from github on a dev/sparkbox? I tried using an ephemeral box but it got some permissions error (i suspect it's IAM role won't let it do request-payer or something like that). I can't do it locally since I don't have great bandwidth now.

copying here s3://REDACTED

@Sheer El Showk yea go ahead

I'd guess we're not so gung ho about pilimi style stuff anymore since we're dropping libgen and pilimi from the next dataset (for legal reasons).

😔

Definitely interested - where would we get 8b google docs from? I'd run an estimate a while ago and only found a couple hundred million.

https://buckets.grayhatwarfare.com/

yeah a bunch of stuff is underrepresented in commoncrawl for some reason so this is definitely not in there

this does seem like one of the higher priority data sources after books+papers! (basically this and extracting more/better out of commoncrawl)

I think its really high priority to figure out how big some of this stuff is and what fraction of it we can actually get. E.g. a rough estimate of 8B docs and 250 toks/doc is 2T tokens?! He also claism 180M files from public s3 buckets, which is also pretty huge

I agree, which is also why it's really high priority to spike out how many extra tokens better CC filtering could be (esp in multiling), which could also be in the one or two trillion range

using the random button it seems like the vast majority of this is imagine though!

this is true for S3 but not Google docs

kipply is registering :loading:

lolwut i found a picture of someone's passport

https://github.com/taewookim/wget_gdrive/blob/main/wget_gdrive.sh

URLs are programatic jfyi

im confused how this website provides gdrive links?

REDACTED

Under shorteners iirc

Hmm they no longer provide that by default 😔 Let me see if I can get the list elsewhere

i've manually looked at maybe 50 text files on buckets now and none of them seem to have usable data, the google docs look really promising though :blobaww: i'm not bothering looking at spreadsheets, 1/3 of them access block me, maybe like 1 in 20 have good data! some examples [x] [x] [x]

this one is awesome!!!! i want all the school assignments

Could we do something like site:.edu https://docs.google.com on common crawl?

yeah of links here with docs.google.com/document , i'd say 1/5 have data but i don't knwo how many such links there are

wdym nova?

How easy is it to estimate 😅

like check if this is in commoncrawl?

Surfacing those links out of commoncrawl yeah

of 700M p5 commoncrawl documents, about 223_013 were gdocs

where p5 is pretty low english filtering but a lot of the docs here seem like high quality multiling

Right, but in the contents of those pages are there any additional gdocs? And I'd guess naively that commoncrawl does a bad job downloading gdocs

wdym additional gdocs?

i expect the format is ok (maybe some extra list tokens) but i can check

Kipply, have you looked at .doc , .docx and .pdf files in greyhatwarfare ? I remember when I was looking into this before (using google search) there were lots of those file types and .doc / .docx were almost always high quality

yeah pdfs on the bucket are really goooood

i think we should ask for premium so we can find out how many docs links and how many pdfs and get a good baseline for how many pdfs/doclinks are good. It's hard to estimate otherwise

I think we can just buy access to the index

:blobaww:

this seems easier to spike than commoncrawl and putting it at higher priority just for that

ok i'm signing up for the PREMIUM stuff (is only 25 euros)

purchased, put login deets in the shared 1pass

for Google docs, can we get sure to scrape via something like docx which preserves the comments? seems like great data for a commenting model

also, did we special case box.com, Dropbox and others?

REDACTED

or like

reasonably good tokens

Hm, do we think there are lots of "reasonably good" tokens thereREDACTED

I guess lots of bad OCR?

REDACTED

REDACTED

hmmm that's a godo point

*good point

let me run the filter on books (trying to get data pipeline to play nice) and i'll push some samples

There is a separate question of "is it good enough that we think we can recover the finch->lark performance hit?". We can likely answer that with evals like the ones Jennifer is running though

yeah it definitely doesn't feel like old-books quality data 😕

i'm going to put up the samples with my current ocr-garbage removal + token density based filtering and leave it for now. Maybe @jennifer can come up with some better filtering ideas but I think I should move on to papers after this

Sounds good - just to get back to the original question do we ahve token estimates for all of it though?

ooh right sorREDACTEDof books REDACTEDicana and universal are not done yet, but i think americana will be likeREDACTEDand my current filtering is false positivey and few false negative)that would bREDACTEDminus dedupe and plus universal. But books tends to get filtered awaREDACTEDronto but I haven't run a proper analysis there yet

i'd be 80% confident that it's 80-200B tokens excluding universal given like up to 20% dedupe rate

Ok, thanks!

ETSI has 24k standards ( https://www.etsi.org)

CEN/CENELEC has another 10k at least, probably more

REDACTEDNot [?] a ton, but its long context which is promising.

https://webstore.ansi.org/

I think we should contact them directly though, the price seems exorbitant individually

Yeah, looks pretty small, and most of them are old handwritten things that we'd need to OCR ourselves and I think it would be hard.

image screenshot attachment; preview not rendered in the export (blank placeholder)
Screen Shot 2022-11-16 at2 MB

I'm a bit confused by The Online Books Page - most of their links seem to send me to Hathi Trust, but not obvious way to download

Similarly confused by DPLA - wasn't obvious to me how to actually download anything from them. If you have a contact there, would be great to intro Tim

For TOBP, it's pretty easy to scrape metadata through the Open Archives Initiative Protocol for Metadata Harvesting, but idk how easy it is to get the non-HathiTrust (extended shelves) items
For DPLA, most items have a URL that goes to a standard ex libris page for download. I don't have a specific contact at ex libris, I assume their sales would be supportive or info@dp.la gets you the DPLA specifically.

Awesome! I'm getting 404 errors on both of those links (probably don't have permission). Can you get a resume and add to Lever? Definitely interested in hiring someone to scrape things, lots of things to scrape 🙂

Links fixed, sorry, was on mobile. I'll ping him and ask for a resume but I added his linkedin to this Lead https://hire.lever.co/candidates/dc817fe9-c96e-4ee8-9217-a90044d2a7b0

My guess is that ANSI and other standards are comparable quality to scientific articles

requesting edit access ❤️

Oops, granted

Nice! Is this sheet updated with the new number of tokens now?

We should figure out weights asap and get sign off from Pranay and Nick if we're planning on doing the update this morning

@yudong can you please check the number of tokens too?

Looks like the number has been updated. LG to me, thanks!

It says (TOKENS NUMBER IS OLD) though. Can we delete that?

i put that bc i put the ones before the most recent dedup

i think

unsure if someone changed it

I think the explore pages aren't updated. @yudong can we regenerate the explore pages?

CORE English Explore

REDACTED

CORE Multilang Explore

REDACTED

do we have a final mix we are using for this?

We have the new explore pages: REDACTED

(I created some new datasets)

you can also get numbers by going data shell -n {regex search of node name} and then selecting which one you want and it will print out the data node in full including n_tokens and n_epoch info

you should be able to open files with this too but i think thats broken for ones with train/test splits

I'm deleting (TOKENS NUMBER IS OLD) because it matches the explore page

Cool I didn't know you could run data shell!

yeah it also lets you run get_node() after opening it to get new ones as it eventually just pops you into interactive ipython shell

What do we mean do we have a final mix we are using for this? Do you mean make a new mix in the code or finalize the mix and weights in the spreadsheet?

i updated the mix in some sense but like have we decided if what on the sheet is final

data shell is cool, thanks, Liz, just to double check, I am using the encoded version of the dataset (instead of the chopped one), is it correct?

no
well
it should be roughly the same
but more accurate would be to grab the token count for chopped and divide by n_epochs

should be about the same though doubt it matters

whats the name of the new chopped nodes

We don't know if it's final yet. I was about to post and get sign off from pranay devi nick, and I wanted to confirm the numbers first

REDACTED and REDACTED

okay they are the same

are they chopped even if they are REDACTED

Thanks for double checking!

sorry i mean are they chopped even though they are called encoded

yeah I was following the older name convention of core

right now we def can't use it the numbers dont add up

wait yudong are they all merged into the REDACTED branch?

I'm glad we're figuring this out now, but would anyone be opposed to moving the update meeting to the afternoon so we have time to get sign off

Re REDACTED branch: not yet, I was basically waiting for the sign-off

realizing less of this is done compared to what I thought:
what we need:

1. make sure we have PRs / all the nodes we need are in the REDACTED branch
2. run 4l test model on new datamix ( i can do this pretty easily)
3. final numbers

https://docs.google.comREDACTED

organization doc

+1 to push the meeting later

yeah sure

Liz, can we pair on this? I only have a rough idea about the steps from this doc @devi sent me:
https://docs.google.comREDACTED

yeah sure

Let's figure out the time for the meeting first, the original schedule is in 11am

I'd also be down to pair w/ you two to learn but I'm enroute to the office. Will be there in ~10. Ff to start w/o me too

Liz I am ready anytime, when it is convenient for you.

whats the plan for REDACTED running out tomorrow? cc @devi @Dickson
dove data runs out in two days. ill be OOO tomorrow but should have all my stuff chopped up by then for new code data without the PRs
(endless exactdedup issues but inshallah it shall work this time)

are you in office yudong

I am working from home today

ill be ready in a min or two, pj will be in agave on 4f

thanks, please take your time

whats the plan for REDACTED running out tomorrow?
PR https://github.com/anthropics/REDACTED Both datasets are committed.

are they in REDACTED?

like the branch i mean

Ah they're not. Let me fix create a branch to merge into REDACTED

https://github.com/anthropics/REDACTED

Merged PR for cc_p8, @Dickson have you transferred the data to gcs and tried loading it in a scan on the REDACTED branch?

Not yet, I can look at that now

i can transfer it over if you want

and was going to scan on REDACTED branch once we get all the new data we want over together. was thinking core + diffs + this at once

Ah sounds good if you do that then. Let me fix the type error.

Let me fix the type error.doesn't matter, it works on master and I already merged this branch

@Dickson check the directory for REDACTED. doesn't look like it has more than REDACTED

that's actually REDACTED

compare with the previous REDACTED which was REDACTED

will it load that fine?

I think it should definitely work for loading data, it might break the tokenslive epoch counting?

data loading is basically based on the metadata.json which is in the root folder and points to all the files with info about how many sequences they have

this has the same structure as the multiple epochs in separate directories we've gotten to work previously, since the model doesn't really track anything about epochs

I believe REDACTED was intended? Original request: https://anthropic.slack.comREDACTED

@Dickson could you take a look at chopping REDACTED

there is no root metadata json but there is for train/

there is no root metadata json but there is for train/that's always the case, we split the metadata per train and test split

figured just checking

anyways moving to gcs now

REDACTED

in the db

thats fixable and outside of the loading i think tho

argh, knew there was something I forgot in the concat implementation

does any other cluster have access to REDACTED or should i just 2pc and run it in oxygen

this should be resolved now, but ftr don't believe we should need 2pc for training on oxygen or reading from that bucket outside of the teal namespace/security class

correct it worked just in oxy without it

Thanks PJ. The more comprehensive deduped is done. Running another scan now 🤞

Excited to see the results whenever they're done!

I'm curious if our PDF dedupe will be good enough to improve the eval results after the last thread

If the the results are the same, I think we can plan to still add it to the mix and improve dedupe in parallel

I am waiting for the larger run to finish (later tonight), for the REDACTED There are still REDACTED (adding @devi as well)

attachment Screenshot 2024-05-15 (280 KB)

I think this is probably fine in that REDACTED

Results of the larger scans are in, they follow the same trend as the previous run. With all the recent discussion, this is now expected. just providing the update to be complete

attachment Screenshot 2024-05-16 (100 KB)

Got it, thanks!

@yudong @pj is our current thinking that REDACTED

That's the current theory, even if we deduped against the mathpix dataset, the current dedup may not be effective enough to filter out the docs in the eval dataset. I am trying to find some direct evidence as well, e.g., by checking whether some papers in the eval set exist in the new CORE dataset

I was checking the eval data, most of the entries have "title", so I just join it with the new CORE data's titles (both to lower-case, no extra processing). Surprisingly there is no overlap on the title between these two sets at all (thanks @rh for calling out the dataset name in the graph, I did not realize it is just yet another dataset 😅 [?]). So maybe the overlap happens in some texts directly? 😕 [?]

By doing a more relaxed matching (check whether the first 50 char of the eval doc title exist in CORE dataset's title), we found one paper "The young massive SMC cluster NGC 330 seen by MUSE" in both mathpix eval and the new CORE data; it is 1 out of 311 entries in the eval set
not sure whether it is significant enough to explain the 2x CM gain though

Can we add them to Dove too!!?

yes we should!

Looks like we received REDACTED scanned books! Started a new thread for processing it. We'll add them to Dove after they're processed

Cool! I think REDACTED books is a small enough sample that it doesn't matter much (maybe REDACTED tokens), though having a simple pipeline for it seems good.

But seems REDACTED

According to email, data is stored in REDACTED I don't have access to the data, so we'll need to make a security ticket for this

lmk if pairing would help!

@kefan are you interested in starting with this and pairing w/ @jennifer?

just tried to ls, yeah, don't have access. which channel should I fire the ticket to?

#it-tickets . Please request access for @ed @brendan @pj @yudong @jennifer too

yeah, that would be fun!

please ask for access from li_eph and from fluorine, not from particular people

the ticket of the it-security was not resolved over night, who should I ping to get it approved?

cc @.security-oncall at the bottom of the ticket and explain that we're blocked

Once you load the data in data pipeline, you should be able to generate an explore page with data explore NODE_NAME

where the node could be the all access right?

some quick updates:

• still trying to get the data access, hope it can resolve very soon, ticket https://anthropic.slack.com/REDACTED
• trying to get a PR to run through the general processing steps. Should be sending for some early feedback soonish.

Also some questions:

• for the dedup, do we do both exactdedup and dedup(which I think is fuzzy dedupe?)
• what are the general workflows to check some samples of the data? @jennifer showed me how to do it in ipython which I was asking if I can do it in colab. But wondering what is the workflow in general

• I can't find the security ticket because it's not linked here, I saw it earlier. We want access from the li_eph and from fluorine clusters, not for specific people like @jennifer corrected me above
• Let's add in dedupe later after we have the other steps of the pipeline working because it fails frequently. Let's make sure the remainder of the pipeline works first. It's a small amount of data, so I expect we can iterate on it quickly
• Load up some data in an ipython notebook or collab, like you said. Our team usually uses ipython

what do li_eph and fluorine mean?

they're both clusters; go/clusters is usually pretty up to date on describing our clusters

I know this is all new! To clarify the initial end goal:

1. We want to look at samples since we've never looked at the data
2. We want to run the data pipeline on the data, which has 3 core steps - load, encode, chop
3. Generate an explore page for these
4. We want to add in the dedupe step too

Thank for the clarification! have a quick PR here for the purpose of the end goal. https://github.comREDACTED still need the chop part but wanna to get some quick suggestions on the dedup part first.

I think it's easiest for you to delete the dedup step for now. We might have a new dedup soon

Example of explore page for a new sample of data: https://anthropic.slack.comREDACTED

Maybe ask @Arian if you can reuse the code for Panama too?

when you say reuse the code, do you mean the processing to get the raw data node to generate a sample? I'm not sure how reusable it'll be since I assume the papers are a different format

i think if i were doing this, the first step would be to run the panama data through a processing pipeline to spit out a parquet file(s): {book_title, text, author, published}

Whats the size of the data? If its very large then you'll need to use dask/ray/spark otherwise just a simple script will do

once we have that we can generate the explore page

It's small - only REDACTED books. We basically want a way to look at the data since we've never looked at it before. I think running the panama data through a data pipeline to get the explore page works. @Arian your explore page looks like it has a few additional thoughts that I thought were interesting if we can get those too. If it's not re-usable, we should just do a new data pipeline

yea fair, if its only REDACTED books then literally REDACTED @kefan once you have that hit me up and i'll help you make an explore page like i did

the fields i said were first thought btw, should preserve any other potentially interesting metadata

I'm seeing this error
REDACTED is it against the policy that I can copy the data into REDACTED where I feel it might make things easier?

where are you seeing this? also prefer REDACTED over REDACTED

I'm trying a pipeline of the books I did that reading books data from REDACTED

the first step seems marking some _SUCCESS into the directory, which might be the cause of the error

ah because it's read-only

yeah you should copy it over

you shouldn't be saving to that bucket anyways right? copy it over into one of our buckets

^ what jennifer said lol

you should copy it over anyway so the pipeline is reproducible

in order to do this, don't set custom_path= and then inside the body just directly copy from REDACTED to REDACTED

I think it's annoying to copy data that we're going to be getting new dumps from every week. What if we just got write perms?

i think that's a very bad idea

Yeah I guess it's data that we're buying and we don't want to accidentally over write

copying is fast and you don't want to fat finger a bad deletion

PJ whats annoying about it? We could just have a simple cron that s5cmds everything over

I thought delete object was a different S3 perm

Yeah, if we have copy job that can copy everything over, let's re-use it. It's not that annoying for me, I'm just trying to reduce overhead for someone that's new and just trying to get the first step of the data pipeline working

copying is easy; shortcutting this is not worth the downsides

I just looked at the data is saved as PDFS so data pipeline can't ingest it anyways. We'll copy it

ah so it needs to go through text extraction

im not sure what you mean by that, we have various data pipeline steps that do pdf extraction

yea thats all i mean

These PDFs are image scans as opposed to having text in the PDF. Do we do OCR as a part of the data pipeline already?

@devi and @liz is helping find an example IRL

ah we don'tmay or may not; how are we planning on doing the OCR, ship to mathpix or run a thing?

So it seems like we have some pdf extraction code in data_pipeline

REDACTED has the raw dataset pdfdrivesample and we should be able to do something similar here

tbc, this is very much a first pass and we can extract a lot more value from the books by doing something more sophisticated than just pdftotext

ok, maybe the way to go is to

• just overwrite the path to save the data to our bucket,
• run the pdftotext,
• then check the output.

yeah try it, lmk how it goes

oh you probably need to tell fishpath what region datamation-scans is, hold on, I'll send you a diff for that

ok if you ever see REDACTED

sample pdf: REDACTED
pdftotext'd: REDACTED

oh @kefan you can post sharable samples of stuff by copying it to REDACTED tho data clusters only have write access to prefix REDACTED

Do we have explore pages for Panama yet? I'd like to confirm that the samples look good and that we're able to extract the data since they're continuing to collect data for us this week

Just synced w/ @kefan. Here's w

Luckily the sample is small and the data pipeline runs fast and works (cc @pranay who asked for a status update on panama)

These are the next steps we discussed so we can finish this up today:
REDACTED

@kefan requested access to doc

Looking at doc

Do you think you can figure out a way to convert PDF to image for this case?

I think it should be one-line with the correct library (see pdf2image, PyPDF2), although you will need to figure out how to install it. We used to be able to install whatever we want, but now it might need to be approved

We're going to working on testing and improving PDF extraction next and we all will want be able to view the PDF as an image for pdf-extraction evals

I know you've been working with a copy of the original dataset. Have we checked if the original data path recently for any updates or new data that may have come in? I'm curious if we have new data that we can run the pipeline on

yeah, there are new datasets coming over the weekend as I mentioned. Also added a figure to track the progress.

on a side note, where should I add the explore pages?

Thanks I didn't see where ever you mentioned that. Feel free to post them here and in your doc. I checked go/data and I only see it for REDACTED Usually we create separate explore pages for chopped and encoded to check if the data looks good, but it's not a blocker since we aren't using this data in prod yet

@pj good idea on converting them into images. Having a colab working pretty nicely to show side by side of one of samples. Also added a figure for daily data trend. With more and more data being processed, we should see the lines being updated. REDACTED

Nice! A few quick thoughts:

• It's nice to have stats! We usually communicate data stats in terms of tokens, so it'd be great if we can update to that metric - tokens received from panama per dump, histogram of tokens per book
• Is number of lines histogram number of lines per book? It's not obvious to me what the first two charts are communicating differently from a skim It'd be excellent if we can add descriptive titles so people can quickly understand the plot before going back to their busy days
• I think it takes a few seconds for the text to show up side by side with the pdf, so I'd add a comment explaining that too
• I think you were running this on a second dump too, so we should rerun data-pipeline and update the numbers in this notebook before wrapping this up. I think you were already working on this

• For the tokens, can you point me to the tokenizer we are using and is it possible for me to use it in notebook? Bytes is normally a good metric but agree we should make it more accurate.
• good suggestions, will update that.
• and yes, running DP on new data dumps.

Btw, the explore page should be in already so please take a look

REDACTED

it works! thank you

Yeah, we have the old explore page, but we should update the explore page after we process the new data dumps too. It'll update the number of tokens and the samples shown

Data partnerships said REDACTED and was curious about the status of our data processing. Do we have results from the newer dump of data yet?

REDACTED please check this for the daily generation

there are some data pipeline issue, mostly I can't get the ray cluster to connect to workers etc

Data partnerships said REDACTED and was curious about the status of our data processing. Do we have results from the newer dump of data yet?would it be possible for me to be in the data partner channel or something? wanna to get to know more of the context.

we have a bit over REDACTED books now, or a bit over REDACTED tokens,

some requests to get things going:

• can you share some dedup code pointer for me to get it going? My plan is to dedup it, REDACTED and run a scan to check if we can have a win on this data
• what is the timeline and process (like 1.xx CM can prove that this is worthy to be added, the trend etc), point of contact, if we want to add it to Teal or Dove?
• when you say REDACTED what is the specific arrangement are we talking about it here, some stats such as expected daily books, metadata of the books (still don't have it), total number we expected, etc, would be appreciated.
• and do we have an internal scheduler that I can config a daily run for the processing and data-pipeline?

• Daily generation stats look good. I'm sure it's on your list to swap number of chars/lines to tokens from yesterday
• If you're having trouble with Ray or anything else, feel free to post in #tokens-onboarding-03-19-devhelp ( @jennifer is the ray superstar)

@pranay

• do you want to rename the channel and remove the date? I think it'd be helpful for tokens new joiners next week too

• go/panama has details about panama and you can ask @pranay about anything else

• Dedupe is still in flux. You can pair w/ @yijie or @brendan to run the new version that's in a branch. This dataset should work REDACTED This shouldn't be a blocker to running the pipeline e2e first because we should know how many tokens we have pre-dedupe anyways

• Timeline depends on the number of tokens, which we will get when you update the explore page and stats page with the newest dump. I'd expect on the order of REDACTED as that's our REDACTED

• REDACTED We'll be receiving new data REDACTED and we should rerun the pipeline with whatever new data we have REDACTED See go/panama

• No, we don't have one. @brendan was handy and hacked something together for crabby

@kefan scheduled a time to pair

Hi @pj @jennifer @devi can you please take a look at this pr REDACTED? it has ran successfully e2e

kefan can you resubmit the PR without the messed up git history

ah the branch has been a while and hard to rebase, recreate a new one

sorry for the delay, here is the a new PR REDACTED

can I have some feedback on the PR? @jennifer @pj @devi

and write down some thoughts on the next steps REDACTED feedbacks are welcomed

sorry, just got back from appointment, reviewing now

Nice, left comments on PR and doc!

thank you both! I'm lacking some of the context of some of the comments. Can you please take another look? thanks!

The OCR from Haiku is so good! Launching an inference, will have a more detailed estimation of the run time on OCR later. cc: @pranay @pj @lzl @tom who might be interested.

Also the scan of a 4x epochs current books data (REDACTED tokens per epoch) just finished. How can I get the downstream evals?

How can I get the downstream evals?cc: @kipply

have loss evals run yet?

and is there a one epoch baseline?

have loss evals run yet?yes, the loss evals ran with names REDACTED. wondering if we have a colab to plot the CM on different evals.

the amount of data is REDACTED or I should REDACTED

all our notebooks are shared in /mnt/notebooks, if you look in REDACTED then there's an example notebook!

we should build a thing that just auto notebooks + generates plots for you though c:

Appreciate it! will get some analysis based on that. Yeah, if we have colab already, seems like a glue job🤣 [?]

I can give it a try?

@pranay @devi this is the first non-text dataset task, so lmk if there's any other text datasets. ( @lzl REDACTED)

The current text datasets are already owned and in progress: REDACTED

If there's a text dataset or other higher priority things I'm happy to take on that instead

Ok! I think this is a good place to start at least for now as I don't think it should take more than a day. I'm happy to pair too. @mitchell has run the most jaxotron scans, so he might have additional notes on the below

Here's a PR with an example scan. The code is simple.

One - You'll want to update the mix to something like this:
REDACTED

Two - update the mixname here.
REDACTED

Three - Run a command like this:
REDACTED

Four - REDACTED

Thanks @pj for the detailed instructions! I'll schedule a pair with @mitchell.

Great, we can take a look and provide comments async if you create a PR beforehand too

SG! Let me give it a try first

go/deepcrawls has a ton of sources that we can triage and prio, as needed 😊

Hmm encountered a weird error: REDACTED when running on both local laptop env and on devbox. I ran this command: REDACTED. I believe I have the right python env py311 on laptop.

Jaxotron runs on TPUs, and the error suggests you're not on a cluster with TPUs. Try use-sodium-beta.

Thanks. Trying

@lzl did you get around this?

Yep, I managed to get a dev box with 16x16 tpus. Running it now

Still running tho

Are you coo tailing the jobs? Are the jobs stepping?

Are you coo tailing the jobsHmm I don't really understand this.
Are the jobs stepping?Unfortunately, it has just failed...

can you post the error

Trying to do it. The stack dump is very long

is it possible to follow the recommendations from the docs and not launch on devbox

OK I think the root error is REDACTED

PJ's example command above REDACTED is meant to be run from your laptop

Likely a bug in my code

yea that looks like a non-jaxotron issue

I see. I think last time I tried to run it on my laptop it says not a TPU cluster

you have to do use-sodium-beta first

that will tell your laptop to launch on the TPU cluster

Oh I see. I thought it's for launching the TPU devbox

Thank let me give it a try

thanks!

I think this means it's working?

attachment image.png

Where can I monitor the jobs?

from your mac you can run coo tail <job-name>

or use go/jaxodash and enter the job name

but you'll still have that data pipeline error

that's not to do with jaxotron, it means that dataset is not there? did you transfer it from s3 to gcs?

Browser says go/jaxodash connection is not secure? Is it expected

yes

Yeah I just commented that dataset out to get the pipeline running first

I didn't know we had to move data, but I'm not surprised. I'm guessing we just use Google STS?

I didn't know we had to move data, but I'm not surprised. I'm guessing we just use Google STS? (edited)yeah, there are some steps in this doc for new datasets for big models, feel free to skip non-applicable steps and substitute relevant bucket (the one used for csp_bucket_override)

you can just change csp_bucket_override and do egress

Right, okay. @lzl for your first scan just change csp_bucket_override. Once that's working, let's figure out the data

i don't actually know what to change that to

but yea that should work

Got it. Just to double check this is the data we want to add the to the mixture right? REDACTED

Yes, that's the data

I see. Should I also "define" the dataset in the codebase? Can't really find the implementation, and I think that's what the previous error message is saying

Also any pointers on where to update csp_bucket_override?

not familiar with jaxotron, but it's generally set on the dataset configuration, not sure how this is handled with the inheritance structure and overrides

in a meeting now but yes

its an argument to the dataset

specifically the JaxotronImageTokenDatasetV2

Ah I see

Thanks!

What's the way to translate the names like REDACTED in the samples html to the actual dataset definition in the codebase?

Aka how can I find out the actual name to put into DatasetMix

REDACTED part means the dataset is defined in REDACTED and the REDACTED means that it's named this either explicitly with a name parameter, or implicitly by assigning a variable/function name to this, or both

the values like REDACTED are the keys in the DatasetMix

Thanks @devi!

Running evals PR by @mitchell from slack which mentioned vqa_pt and chartqa_pt, those are what you want to run too

#eval-toolbox-users is another slack channel for help related to evals

Replace sandbox/sandbox/mitchell/snapshot_lists/eider_scan.txt with just your model name
REDACTED

@lzl REDACTED

Eval done: REDACTED is the baseline model that REDACTED (note that this run also doesn't have REDACTED dataset, so weighting might be different here), and REDACTED is the one with the dataset. Overall, REDACTED. Let me know if this makes sense, or I should have a proper baseline using the computer use dataset

attachment image.png

any idea why REDACTED

These evals REDACTED

can you send loss throughout training?

there should also be loss evals throughout training we can look at

but actually idk how useful those will be if training was entirely done on the source

Hmm not sure why REDACTED Could it be REDACTED?

What's the reasonable range for those tasks? Not super familiar with these tasks

Training loss looks REDACTED

attachment image.png

Test loss during training is REDACTED

attachment image.png

can you link loss plots with spikes?

Hmm not sure how to share a specific plot from go/jaxodash? Is screenshot ok?

@rh what were you referring to?

This is the train loss REDACTED

attachment image.png

The uptime stats. The job under the question is REDACTED

attachment image.png

This PR has all the changes I made

i think something about the data is REDACTED

REDACTED

REDACTED

REDACTED

Here's an example of previous results. It looks like REDACTED I just saw your results in your notebook too. Maybe try running the evals starting w/ the REDACTED

Eval run comparing with and without synthetic charts on different model sizes. Generally it seems like REDACTED

attachment newplot.png

newplot (3).png

attachment newplot (3).png

@lzl can you please share your visualization links here too?

linking jaxotron thread about passing a seed during temperature sampling:
REDACTED

relevant thread

Vis links:

• REDACTED

REDACTED

Only REDACTED model shows REDACTED

attachment image.png

Maybe post that in the above relevant thread?

@rh @lzl and I paired and decided to table this here b/c there's more work for the evals and an mm person should probably own that cc @ed who might pick it up. @lzl you should plan to write this up and maybe move onto pdfs

Is it prev thread you linked to above?

Yeah or I think writing down your results in a doc and letting multimodal people see and comment on it might be useful too

Sounds good. I'll do a doc which seems to be cleaner than copying everything over to another thread

Wrote a doc for it REDACTED

Nice! Left a few comments

Considering this thread of work done for now although there are more threads to pull on in the future!

How good of an understanding do we have as to REDACTED? Is it REDACTED

@jennifer REDACTED
enjoy a giant thread about REDACTED

Thanks for starting the thread, PJ! I am also wondering, as @rh brought up, what is the testset (especially the mathpix test set) we are using? Is it all in unpaywall.mathpix_outputs_filtered_deduped?

isn't it the aug23 mathpix in the chart title here? REDACTED

REDACTED

REDACTED

REDACTED

Thread here. I think we have REDACTED cc @ed can we REDACTED

They didn't ask for this but I think we REDACTED already? REDACTED

also lower priority but REDACTED

Also do we want REDACTED

(running REDACTED now)

New dedupe methodology by @rh:
REDACTED

Other things tried:

• REDACTED

This is the deduplication methods we're currently using for PDFs. It's implemented differently on ray and dask, so different datasets are using different methods.

• REDACTED

Other deduplication methods in the codebase:
REDACTED

Future Work
REDACTED

yeah i talked to devi about this and realized i didnt actually check that lol.

yall pj is trying to post the initial thread still