0

I have been going through the GitHub repository for SparkNLP and PySpark and my objecive is to be able to add another layer to BertSentenceEmbeddings.

Pyspark seems to eventually call the Bert implementation written in Scala (perhaps?).

Being a python developer, is there a pythonic way to access last layer of model via pyspark (or python) to add a penultimate layer to create a new / update the model?

Currently, I'm trying to manipulate the final embeddings I obtained from spark pipeline through dataframe manipulation. But I'm interested in making it a part of the model output to streamline processes.

Note: Using pyspark 3.3.1

Improve this question
Related questions
5
Is it possible to call a python function from Scala(spark)
1
Spark not executing tasks
4
Spark Python Pyspark How to flatten a column with an array of dictionaries and embedded dictionaries (sparknlp annotator output)
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.