Back to all discussions

  •   about 2 years ago

JSON mode

Can Arctic reliably generate its output in JSON format only?

  • 6 comments

  • Manager   •   about 2 years ago

    Hey Balaji! If I understand your question correctly, you're looking to constrain the output of the LLM to JSON only and ensure it's valid JSON.

    If you are working with the Replicate API endpoint, I believe you should be able to do this directly via the Replicate API like so: https://replicate.com/changelog/2023-03-20-get-model-input-and-output-schemas-via-the-api

    I believe you can also do the same thing in Snowflake Cortex if you are using Cortex for this challenge:
    https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex

    If you're not using either API, there are some libraries that you can use to achieve this, e.g.:
    https://github.com/outlines-dev/outlines
    https://github.com/guidance-ai/guidance

    Let me know if this doesn't answer your question and happy hacking! :)

  •   •   about 2 years ago

    Thank Anna!

    I clicked through the links and I don't think that is what I am looking for. Here is an example of what I am looking for

    on https://arctic.streamlit.app/ if I enter the prompt "create 5 options for cat names in JSON format. I want the keys to be 'option1', 'option2', 'option3', 'option4' and 'option5'"
    As expected I get the desired response: { "option1": "Whiskers", "option2": "Mittens", "option3": "Smokey", "option4": "Luna", "option5": "Garfield" }

    I've tried it using the replicate API as well with similar results.

    However, this does not seem to work reliably for longer system messages (~1000 tokens long)

  • Manager   •   about 2 years ago

    Ahh I see what you're saying! Let me check with the AI team real quick. BRB

  • Manager   •   about 2 years ago

    Hey there Balaji! Thanks for your patience. It looks like this is a limitation of the model itself in that it gets less good at outputting structured data as the response gets longer -- the Streamlit app you link is designed with some guardrails that kick in at that scale. My suggestion would be to either focus your project on an application that doesn't run into this as a requirement, or focus your project on solving for creating some of these guardrails as your actual submission-- that could be a really valuable contribution to the open source project itself, if you're up for the challenge! :)

  •   •   about 2 years ago

    Thank you for the response!

  •   •   about 2 years ago

    Thank you for your response Anna.

    I ended up building guardrails as part of my actual submission -- https://arctic-analytics.streamlit.app/

Comments are closed.