Function level tokenizer #4

Open

opened

on Jan 12, 2023

Hi, great repo. Could you share the Normalizer and Java Parser code for function_level to regenerate token list similar to your function_normalized_tokens and function_tokens in the original processed dataset ?

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests