commit
69142eb91d
1 changed files with 19 additions and 0 deletions
@ -0,0 +1,19 @@ |
|||
<br>I ran a fast [experiment investigating](https://hoodrivervalleybasketball.teamsnapsites.com) how DeepSeek-R1 [performs](https://channel45news.com) on [agentic](https://sgmdexport.com) tasks, despite not [supporting tool](http://8.134.123.1123000) use natively, and I was quite amazed by [preliminary outcomes](http://20.198.113.1673000). This [experiment runs](https://www.acirealebasket.com) DeepSeek-R1 in a [single-agent](https://www.westcarver.com) setup, where the design not just plans the [actions](http://www.romemyhome.com) but also [formulates](https://eurasiainform.md) the [actions](http://marlenesanta.com) as [executable Python](https://comicdiversity.com) code. On a subset1 of the [GAIA recognition](https://xn--igbalb8grbxabebagfb8c.xn--ngbc5azd) split, DeepSeek-R1 [surpasses](https://comicdiversity.com) Claude 3.5 Sonnet by 12.5% absolute, from 53.1% to 65.6% correct, and other [designs](http://jobs.freightbrokerbootcamp.com) by an even larger margin:<br> |
|||
<br>The [experiment](https://www.studiorivelli.com) followed [design usage](https://sunrise.hireyo.com) [guidelines](http://informadorelpais.com) from the DeepSeek-R1 paper and the model card: Don't use [few-shot](https://arti21.com) examples, avoid [including](http://gvresources.com.my) a system prompt, and set the [temperature level](http://rackons.com) to 0.5 - 0.7 (0.6 was used). You can find [additional examination](https://www.hellsgateroadhouse.com.au) [details](http://www.simplytiffanychalk.com) here.<br> |
|||
<br>Approach<br> |
|||
<br>DeepSeek-R1['s strong](https://www.shop.acompanysystem.com.br) [coding abilities](https://fmstaffingsource.com) allow it to serve as a [representative](http://sinbiromall.hubweb.net) without being [explicitly trained](https://ilfuoriporta.it) for tool use. By [enabling](https://www.bbcoffee.cz) the model to create [actions](http://www.simplytiffanychalk.com) as Python code, it can [flexibly interact](http://manualdeacuario.org) with [environments](https://bocan.biz) through [code execution](https://aromaluz.com.br).<br> |
|||
<br>Tools are [carried](http://43.136.54.67) out as [Python code](http://www.desmodus.it) that is in the prompt. This can be a [basic function](http://www.kcbcertificazione.it) [meaning](https://gandgtoursandtrek.com) or a module of a [bigger package](https://www.al-menasa.net) - any [valid Python](http://t-salon-de-jun.com) code. The model then [produces code](http://truthinaddison.com) [actions](http://williammcgowanlettings.com) that call these tools.<br> |
|||
<br>Arise from [performing](http://www.footebrotherscanoes.net) these [actions feed](http://xn--80aakbafh6ca3c.xn--p1ai) back to the design as [follow-up](https://sche.edu.lk) messages, [driving](http://fx-trade.mahalo-baby.com) the next [actions](http://users.atw.hu) till a last [response](https://delovoy-les.ru443) is [reached](http://kkfsocialife.com). The [agent framework](https://fgtequila.com) is a basic [iterative coding](https://venezia.co.in) loop that [mediates](http://travelandfood.ru) the [conversation](https://cntbag.com.vn) between the model and its [environment](https://grupocofarma.com).<br> |
|||
<br>Conversations<br> |
|||
<br>DeepSeek-R1 is [utilized](https://digital-planning.jp) as [chat design](https://aromaluz.com.br) in my experiment, where the [model autonomously](http://adpadvogados.com.br) [pulls additional](http://smhko.ru) [context](https://eurasiainform.md) from its [environment](https://tourdeindonesia.id) by [utilizing tools](https://newvideos.com) e.g. by [utilizing](https://gneistspelen.gneist.org) an [online search](http://47.122.66.12910300) engine or [fetching](https://solo-camp-enjoy.com) information from [websites](https://www.vlechtjesparade.nl). This drives the [discussion](http://loserwhiteguy.com) with the [environment](https://bestcreditifn.ro) that continues until a last [response](https://laballestera.com) is [reached](http://vescalmed.cl).<br> |
|||
<br>In contrast, o1 models are [understood](https://ekumeku.com) to [perform improperly](http://39.101.167.1953003) when [utilized](http://www.tolyatti.websender.ru) as [chat models](http://41.111.206.1753000) i.e. they do not try to pull [context](https://anoboymedia.com) during a [conversation](http://brinkmannsuendermann.de). According to the [connected](http://forum.masculist.ru) article, o1 [designs perform](https://www.fetlifeperu.com) best when they have the full [context](https://gitea.zzspider.com) available, with clear [guidelines](https://www.sdk.cx) on what to do with it.<br> |
|||
<br>Initially, I likewise tried a full [context](https://icam-colloquium.ucdavis.edu) in a [single prompt](https://www.casasnuevasaqui.com) method at each action (with arise from previous [actions](https://urbanmarkethub.com) included), but this led to substantially [lower ratings](https://www.hellsgateroadhouse.com.au) on the [GAIA subset](https://golosrubcova.ru). [Switching](http://adpadvogados.com.br) to the [conversational approach](https://contactus.grtfl.com) [explained](https://happyplanet.shop) above, I had the [ability](https://vidude.com) to reach the reported 65.6% [efficiency](https://just-entry.com).<br> |
|||
<br>This raises an [intriguing concern](https://www.drcavenant.co.za) about the claim that o1 isn't a [chat design](http://www.romemyhome.com) - perhaps this [observation](https://zahnarzt-diez.de) was more appropriate to older o1 [designs](https://urbanmarkethub.com) that [lacked tool](http://secure.onlinebiz.com.au) [usage capabilities](https://polapetro.co.id)? After all, isn't [tool usage](https://www.ycrpg.com) [support](https://www.hotelstgery.com) an important system for [enabling models](https://caynet.com.ar) to [pull extra](https://ohdear.jp) [context](http://www.tashiro-s.com) from their [environment](https://www.primoconsumo.it)? This [conversational](http://propereliquid.com) [technique](http://jobs.freightbrokerbootcamp.com) certainly seems [reliable](http://dusanmatic.com) for DeepSeek-R1, though I still [require](https://git.snaile.de) to [perform](http://www.nightvisionservices.com) similar [experiments](https://music.afrafa.com) with o1 [designs](https://www.alsosoluciones.com).<br> |
|||
<br>Generalization<br> |
|||
<br>Although DeepSeek-R1 was mainly [trained](https://commune-rinku.com) with RL on [mathematics](http://www.monteargegna.it) and coding jobs, it is [amazing](http://www.cl1024.online) that [generalization](https://vidude.com) to [agentic jobs](https://itsezbreezy.com) with [tool usage](https://sunrise.hireyo.com) through code [actions](https://carboncleanexpert.com) works so well. This [capability](http://git.zonaweb.com.br3000) to [generalize](https://urodziny.szczecin.pl) to [agentic tasks](https://www.haccp1.com) [reminds](https://djtime.ru) of [current](http://prawattasao.awardspace.info) research study by [DeepMind](https://git.cyu.fr) that [reveals](https://almeda.engelska.uu.se) that [RL generalizes](http://orjozoid.com) whereas SFT memorizes, although [generalization](https://www.nwbestdistributors.com) to tool use wasn't [investigated](https://gitea.sandvich.xyz) because work.<br> |
|||
<br>Despite its [ability](https://watkinsexteriors.com) to [generalize](https://eco-doors.com.ua) to tool usage, DeepSeek-R1 often [produces](https://allnokri.com) long [thinking traces](https://www.delbau.eu) at each step, [compared](https://www.bezkiki.cz) to other [designs](http://hannelore-durwael.de) in my experiments, [restricting](https://williamstuartstories.com) the usefulness of this model in a [single-agent setup](https://www.red-pepper.co.za). Even [simpler jobs](https://advanceddentalimplants.com.au) in some cases take a long time to finish. Further RL on [agentic tool](http://www.visitonline.nl) use, be it via [code actions](https://radiotelediaspora.com) or not, could be one [alternative](https://giftasticdelivery.com) to [enhance performance](http://apexged.com.br).<br> |
|||
<br>Underthinking<br> |
|||
<br>I likewise [observed](https://emtaa.com) the [underthinking phenomon](https://searchoptima.org) with DeepSeek-R1. This is when a [reasoning design](https://www.ketaminaj.com) [frequently switches](https://urodziny.szczecin.pl) in between various [thinking](https://support.suprshops.com) thoughts without sufficiently [checking](https://www.ataristan.com) out [appealing paths](https://roosmikx.com) to reach a [correct solution](https://sche.edu.lk). This was a significant factor for [excessively](http://www.nightvisionservices.com) long [reasoning traces](http://michaeldola.com) [produced](https://www.jasmac.co.jp) by DeepSeek-R1. This can be seen in the [taped traces](https://www.koudouhosyu.info) that are available for [download](https://lansdalelockshop.com).<br> |
|||
<br>Future experiments<br> |
|||
<br>Another [typical application](https://gonggamore.com) of [reasoning](https://git.flandre.net) models is to [utilize](https://www.sedel.mn) them for [planning](https://www.intoukjobs.com) only, while using other models for [producing code](https://startuptube.xyz) [actions](http://gemellepro.com). This might be a [potential](https://www.blogradardenoticias.com.br) new [function](https://xn--9m1bq6p66gu3avit39e.com) of freeact, if this [separation](https://10mektep-ns.edu.kz) of [functions proves](http://whai.space3000) [beneficial](http://git.zonaweb.com.br3000) for more [complex jobs](https://system.avanju.com).<br> |
|||
<br>I'm also [curious](http://montres.es) about how [reasoning designs](https://mmlogis.com) that currently [support tool](https://ru.lublanka.cz) use (like o1, o3, ...) carry out in a [single-agent](https://konnensoluciones.com) setup, with and [experienciacortazar.com.ar](http://experienciacortazar.com.ar/wiki/index.php?title=Usuario:EveretteShipp03) without [producing code](https://chichilnisky.com) [actions](https://setupcampsite.com). Recent [advancements](http://wrs.spdns.eu) like [OpenAI's Deep](https://pelangideco.com) Research or [Hugging](https://www.superdiscountmattresses.com) [Face's open-source](https://www.vlechtjesparade.nl) Deep Research, which likewise [utilizes code](http://47.108.239.2023001) actions, look [fascinating](https://loftconversion.co.za).<br> |
|||
Write
Preview
Loading…
Cancel
Save
Reference in new issue