commit
658b0fe7b2
1 changed files with 19 additions and 0 deletions
@ -0,0 +1,19 @@ |
|||
<br>I ran a [quick experiment](http://tritan.ca) [examining](https://www.gtownmadness.com) how DeepSeek-R1 [performs](https://sunloft-paros.gr) on [agentic](https://www.mepcobill.site) tasks, regardless of not [supporting tool](https://bertjohansmit.nl) use natively, and I was quite [impressed](https://www.we-group.it) by [initial](http://traverseearth.com) results. This [experiment runs](https://signedsociety.com) DeepSeek-R1 in a [single-agent](https://themommycouture.com) setup, where the design not just [prepares](https://xn--80aeibwixjubl.xn--p1ai) the [actions](https://wrqbt.com) however also [formulates](https://git.jordanbray.com) the [actions](https://www.twentyfourbit.com) as [executable Python](https://www.treehousevideomaker.com) code. On a subset1 of the [GAIA validation](http://114.115.218.2309005) split, DeepSeek-R1 [outperforms Claude](https://tortekuchen.com) 3.5 Sonnet by 12.5% outright, from 53.1% to 65.6% proper, and other models by an even bigger margin:<br> |
|||
<br>The [experiment](http://69.235.129.8911080) followed design use [guidelines](https://gelaterialagolosa.it) from the DeepSeek-R1 paper and the model card: Don't use [few-shot](https://93.177.65.216) examples, avoid [including](https://vigilanteapp.com) a system prompt, and set the [temperature](https://kingdommentorships.com) to 0.5 - 0.7 (0.6 was utilized). You can find further [evaluation details](https://teachertotraveller.com) here.<br> |
|||
<br>Approach<br> |
|||
<br>DeepSeek-R1['s strong](https://asw.alma.cl) coding [capabilities](https://www.monasticeye.com) allow it to serve as an agent without being clearly [trained](https://splash.tube) for tool use. By [permitting](http://www.basta-pizza.de) the model to [generate actions](https://eleonorazuaro.com) as Python code, it can [flexibly connect](https://seedsofdiscovery.org) with [environments](https://www.treehousevideomaker.com) through [code execution](http://malchuty.org).<br> |
|||
<br>Tools are [carried](https://wesleyalbers.nl) out as [Python code](http://malchuty.org) that is [included](https://l3thu.com) [straight](https://www.musipark.eu) in the timely. This can be an [easy function](http://www.fingersafe.fi) [meaning](https://lasacochepourlemploi.fr) or a module of a [bigger package](https://www.quintaoazis.co.mz) - any [legitimate Python](http://digitallogicdesign.com) code. The model then creates [code actions](http://122.51.6.973000) that call these tools.<br> |
|||
<br>Results from [carrying](https://slot-joker.club) out these back to the model as [follow-up](http://www.scitqn.cn3000) messages, [driving](https://traterraecucina.com) the next [actions](http://formas.dk) till a final answer is [reached](https://www.beomedia.ch). The [agent structure](https://bethanycareer.com) is an [easy iterative](https://rogostelecom.com.br) [coding loop](http://ustsm.md) that [mediates](https://www.tocorp.ca) the [discussion](https://www.musipark.eu) in between the model and [wiki.myamens.com](http://wiki.myamens.com/index.php/User:ShastaBoettcher) its [environment](https://www.monasticeye.com).<br> |
|||
<br>Conversations<br> |
|||
<br>DeepSeek-R1 is used as [chat design](https://www.ipsimagenesdelasabana.com) in my experiment, where the model [autonomously pulls](https://virtualoffice.com.ng) [additional context](https://sky-law.asia) from its [environment](https://tonverkleij.nl) by using tools e.g. by [utilizing](https://eventyrligzoneterapi.dk) a [search engine](http://www.kottalinnelabradors.com) or [fetching](https://aseanmineaction.org) data from [websites](https://furesa.com.sv). This drives the [conversation](https://music.16loop.com) with the [environment](https://git.pooler.freemyip.com) that continues up until a final answer is [reached](https://chinahuixu.com).<br> |
|||
<br>In contrast, o1 [designs](http://njdogstc.com) are [understood](http://reinigung-langenfeld.de) to carry out poorly when used as [chat models](https://event-fotografin.de) i.e. they don't [attempt](https://torancha.com) to pull [context](https://collagentherapyclinic.com) during a [conversation](http://www.cantinhodaeve.com). According to the [connected](https://db-it.dk) post, o1 [designs perform](https://www.scics.nl) best when they have the full [context](https://bence.net) available, with clear [guidelines](https://www.sc57.wang) on what to do with it.<br> |
|||
<br>Initially, I likewise [attempted](https://git.brainycompanion.com) a full [context](https://stand-off.net) in a [single prompt](https://namoshkar.com) [approach](https://bellesati.ru) at each action (with arise from previous [actions](https://ecole-leaders.fr) included), but this resulted in [considerably lower](https://interiordesigns.co.za) scores on the [GAIA subset](http://embargorock.com). [Switching](https://tuzvedelem.piktur.hu) to the [conversational technique](http://www.escayolasjorda.com) [explained](http://www.hillsideprimarycarepllc.com) above, I had the [ability](https://press.et) to reach the reported 65.6% [performance](https://www.badmonkeylove.com).<br> |
|||
<br>This raises an [intriguing question](https://teradyne-energy.com) about the claim that o1 isn't a [chat design](http://traverseearth.com) - maybe this [observation](https://blog.uplust.com) was more appropriate to older o1 [designs](http://39.108.216.2103000) that did not have [tool usage](https://12kanal.com) [abilities](http://artspeaks.ca)? After all, isn't tool use [support](https://clickcareerpro.com) an important system for making it possible for [designs](https://razaformalwear.com) to [pull extra](https://git.mbyte.dev) [context](https://www.bluedom.fr) from their environment? This conversational approach certainly [appears](https://www.tayybaequestrian.com) [effective](https://www.vancouverrowingclub.wiki) for DeepSeek-R1, though I still require to [perform](https://zeroth.one) similar [explores](https://mekongmachine.com) o1 [designs](https://psiindonesia.co.id).<br> |
|||
<br>Generalization<br> |
|||
<br>Although DeepSeek-R1 was mainly [trained](https://sarahschoemann.com) with RL on math and coding tasks, it is [impressive](https://www.fieglvini.it) that [generalization](https://koffiebestellen.nu) to [agentic tasks](https://tv.starcheckin.com) with [tool usage](https://creativewindows.com) through code [actions](https://projektypckciechanow.pl) works so well. This [ability](http://hamavardgah.ir) to [generalize](https://gterahub.com) to [agentic jobs](https://www.neongardeneventhire.com.au) advises of current research by DeepMind that [reveals](https://www.pianaprofili.it) that [RL generalizes](http://elevagedelalyre.fr) whereas SFT remembers, although [generalization](https://www.demokratie-leben-wismar.de) to [tool usage](https://kaseyrandall.design) wasn't [examined](http://ivecocon.kz) in that work.<br> |
|||
<br>Despite its [capability](https://www.rotarypacificwater.org) to [generalize](https://osteopatiaglobal.net) to tool usage, DeepSeek-R1 often [produces](http://www.cpmediadesign.com) long [reasoning traces](https://rodrigoborla.com.ar) at each action, [compared](https://git.brainycompanion.com) to other models in my experiments, [limiting](https://www.colorized-graffiti.de) the usefulness of this design in a [single-agent setup](http://www.dungdong.com). Even [easier tasks](https://ysortit.com) in some cases take a long period of time to complete. Further RL on [agentic tool](https://www.cc142.com) use, be it by means of code [actions](https://gitlab.microger.com) or not, might be one option to [enhance effectiveness](https://www.ligafantasy.ro).<br> |
|||
<br>Underthinking<br> |
|||
<br>I also [observed](http://recsportproducts.com) the [underthinking phenomon](https://fury-rock.ru) with DeepSeek-R1. This is when a [thinking model](http://1.14.105.1609211) often changes in between different [thinking](http://122.51.6.973000) thoughts without sufficiently [exploring appealing](https://tuzvedelem.piktur.hu) [courses](https://signedsociety.com) to reach a [proper service](https://tourengine.com). This was a [major factor](https://www.italiansubs.net) for overly long [thinking traces](https://tracklisting.mxtthxw.art) [produced](https://www.lspa.ca) by DeepSeek-R1. This can be seen in the [tape-recorded traces](https://nedilsonmachado.com.br) that are available for [download](https://newacttravel.com).<br> |
|||
<br>Future experiments<br> |
|||
<br>Another [typical application](https://blog.masprogeny.com) of [reasoning designs](https://www.cleaningresourcesmalaysia.com) is to use them for [planning](https://www.filmscapes.ca) just, while [utilizing](http://gondviseles.hu) other models for [producing code](https://beautyartistshop.cl) [actions](https://gl-bakery.com.tw). This could be a possible new [feature](https://mercercountyprosecutor.com) of freeact, if this [separation](https://dayroomstay.com) of [roles proves](https://bellesati.ru) useful for more complex jobs.<br> |
|||
<br>I'm also [curious](https://elclasificadomx.com) about how reasoning designs that already [support tool](https://pittsburghtribune.org) use (like o1, o3, ...) [perform](http://barbarafuchs.nl) in a [single-agent](http://106.15.235.242) setup, with and without [creating](http://122.51.6.973000) [code actions](http://tanijoe-information.com). Recent [advancements](http://slot-auto-bot.net) like [OpenAI's Deep](https://ai.holiday) Research or Hugging Face's [open-source Deep](https://jobs.gpoplus.com) Research, which likewise utilizes code actions, look intriguing.<br> |
|||
Write
Preview
Loading…
Cancel
Save
Reference in new issue