Touch: It's really useful when the action is generated.
It's a touch-and-go game.
In many of our previous posts, we've all mentioned that filters plug touch directly into larger models, often with negative effects.
So more and more teams are starting to work on the same thing: how to do better.It's a touch of "access".
Recently, TF2-Steering, a team from Zhejiang University, Shanghai Institute of Innovation, and others, proposed a new entry point: changing the direction of the touch at the moment the action is generated.
Touch experts predict an increase in speed at each step of the flow matching, moving the motion expert's speed field to the right direction of contact.
It's not.On the basis of Sharpa's handheld machine experiments, the average progression success rate was 71.5%, up 9.1 percentage points from 62.4% for the T-Rex; the injection task increased from 56% to 94%.
The question is where to go next.
Two questions that haven't been answered seriously
The study opens with two questions:
First, how do you integrate touch into a visual-dominated basic strategy?
Touch signals are very rare in time (contact occurs only in a small portion of the task) and the amount of data is much smaller than visual. Directly entering the input, the model either ignores it or is distracted by it.
Second, is touch just feedback, or is it just a prediction?
People do two things with precision: look at the object to predict how much force to use and how to hold it; and then, after real contact, feel it online. The current job is either to do only reactive feedback (T-Rex) or to make only future tactile predictions, which are rarely stuck in the same causal chain.
The answer to TF2-Steering is: both, and all, get to the "how the action is generated" step.
There are two approaches that have short boards.
Currently, the main means of isolation are leftover passageways and gate control.
The problem with the residual pathway is that it can only be modified after the action has been generated and cannot pull back a sequence of actions that went wrong on its own;
The door control maintains the main handle, but the learned door may decrease it before the tactile feature reaches the action head, making the tactile contribution weak and unstable.
Related experiments show the same structure, only after the modification is moved to generation, the average drop from 72.3% to 61.3%, and 61.7% without touch is basically flat.
It's like a white dress.
To make the touch a deviation in the speed field
Don't change the engine, just change the direction
The flow matching motion is generated, essentially along a speed field from the noise score to the motion. The practice of TF2-Steering can be summarized in one sentence:
Final speed = original speed + touch adjustment.
TF2-Steering architecture overview. Video and touch world models provide low frequency planning pioneers, motion and touch experts perfect movements at high frequency through the increased speed they learn; one-way attention unravels the primitive world of action and touch input.It's a great way to get started.
In other words, a basic strategy is a basic strategy whose output is "roughly where it should go" that does not change; the touch is only responsible for superimposing a local deviation in this direction.
The direct benefit of this design is that motion experts can be completely unaware of the presence of the touch without the touch sensor, running only v_base, and the model is usually available. This is much more important in engineering than "must have touch to run".
There are two key constraints on training to prevent "change direction" from turning into "change ability":
One-way attention: touch specialists can see the characteristics of motion specialists, and vice versa.
Stop gradient: Losses of touch correction are not retransmitted to update motion experts; motion experts are trained only by their own motion loss.
At the same time, the output of the modifier head is exactly 0, so that the pre-training strategy is not confused at the beginning.
Tactile experts: two-way experts, weight is not one
There is a small MoE inside the touchscreen:
One way is to process the future touch of the forecast, one way is to process the real-time touch, each independently of the FFN, and then use the sigmoid overweight convergence, the two weights do not require and are equal to 1.
This detail is interesting: the two-way information is not duplicate, but can be simultaneously weighted and compressed.
Tactile world model: only by characteristics, not by decoding
The touch world model is built on LTX-Video, with about 500M parameters, cutting off multi-point intersection attention, leaving only about a third of the layer number, and the input is the current touch map, predicting future touch.
Interestingly, when reasoning, it doesn't decode the tactile graph, but instead directly transfers the last layer of characteristics to the tactile expert.
This is because decoding is both time-consuming and missing information, and the downloadable content is "tasking-related contact in context", not a good-looking predictive chart.
Image left: Pure vision and visual acuity touch pre-training data. Middle: touch experts use touch world model cache predictions and real-time touch feedback predictions to increase speed, modify motion experts' flow matching speeds; video world models and motion experts can still be used independently when there is no touch input. Right: task-related post-training, and evaluation in simulation and the real world.It's a great way to get started.
Double rate: 5 Hz thought, 30 Hz changed
One system, two frequencies
Low frequency: 5 Hz
Video World Model (LTX-Video, 2.3B) + Tactile World Model (500M) is responsible for long-range visual and tactile precognition, with one block of action only reasoned once and then fixed during the entire execution process.
High frequency: 30 Hz
200M) + Touch Expert (200M) takes the latest touch feedback and reproduces the remaining actions that have already been executed.
In real-machine experiments, this set of high-frequency tactile feedback comes from the 22 degrees Sharpa hand on a dual UR5e mechanical arm, with visual observation provided by the top D455 and the wrist D405.
It's a real robot platform and demonstration collection.It's a great way to get started.
When training, a delay d (delaying the arrival of simulated tactile feedback) is taken at random to calculate the correction loss only within a window after d, and gives each action a relative time-coding ρ=i−d, allowing the tactile feedback and the "how many steps it should correct" to match up.
In logic, k can be tuned, which is the spindle between the response rate and the computational cost.
Additionally, in video model training, the flow matching noise ratio is set to 1, equivalent to one-step noise, and only one forward reasoning is required; an action block results in 240 ms on a single RTX 4090.
Three stages are needed.
Touch world model pre-training: OpenNeoData: 30%) + T-REX: 40%) + FTP-1-Sharpa: 30%, 50,000 steps, batch 1024.
Tactile speed correction pre-training: four modules + Q-Former end-to-end connectivity, tactile data accounting for 90%, and mixing 10% of non-tactile samples maintaining the main stem generalization.
After-task training: Simulation and adaptation of tasks to real-machine data.
Chart: The role of touch pre-training. (a) The success rate of four real-world operations in progress under pre-training with/without touch. (b) The qualitative contrast of touch prediction: from left to right, respectively, for visual observation, true-value touch observation, and touch prediction given by models with/without touch pre-training.It's a great way to get started.
Pre-training is not an option. Skipping two stages of tactile pre-training, training directly from the world-action master to do the task, the success rate of the four real-machine missions all decreased.
The results of the experiment
Simulation: Average rises, but increases are highly concentrated
In the table below, the average is 68.3%, which is 10.6 percentage points higher than the strongest FTP-1 (57.7%).
Table of UniVTAC simulation results. Success rates for six missions: % Avg. for the macro-average; coarse and bottom-line for each column: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +It's a great way to get started.
The increase was almost entirely from Insert Hole: 94.0% to 56.0%, a task that contributed 38 points.
In contrast, the Put Bottle is only 44.0%, the FTP-1 is 74.0%; the Pull Out Key is 34.0%, or less than ACT's 46.0 and VITaL's 47.0; the Lift Can is 60.0, or less than π0.5's 74.0.
This indicates that touch modification is good at the insertion class task that determines the success of the geometry of the touch modification, which is not superior to the task of grabbing and moving the touch modification, and even lags behind because the modification interferes with the correct action.
The average value overshadows the bias on this type of task.
It's 71.5%, but not a "success rate".
The entire real-machine experimental platform is equipped with Sharpa's tactile hand, with two UR5e mechanical arms to complete all six operations.
Among them, the hand-built fabric touch sensors are capable of continuous output of high-density contact signals, supporting such fine, continuous contact interactions; andThe 22 active degrees of freedom and 1:1 man-to-man ratio also provide space for multi-finger collaboration tasks such as bottle caps and Larachains.
Based on real-world experiments on this platform, the average success rate for six missions was 71.5%, up 9.1 percentage points from T-Rex's 62.4%.
See the following assignments: bottle cap 67 to 58, drop tube transfer 66 to 46, wipe plate 76 to 61, draw chain 60 to 56, split cup 90 to 87;
The real-world results -- the success rate of 20 trials per mission, the Avg. is the macro average of six trials.It's a great way to get started.
Only fruit plates 70.0 are lower than T-Rex's 72.5 and FTP-1's 75.0.
The reason for this is that the packaging is based on the marginal value of capturing and placing, which would have been lower than the marginal value of touch.
The real machine used here is the progression success rate (PSR), which gives a score of 0 to 1 per 20 trials per milestone, so 71.5% is not equal to "71.5% of the trials completed the task", it's more like "seventy percent of the tasks completed on average".
For example, pull the chain: raise the bag 0.2, start pulling but not pulling 0.6, pull 1.0, take the average.
Drawings of real-world operational missions and examples of execution sequences. (a) Bottle caps and drag chains, illustrated for touch visualisation. (b) Fruit plates, wiping plates, separating cups, droplets for transfer of liquid, bottle caps, drag chains, scenes of six missions.It's a great way to get started.
The three counter-intuitive conclusions of the dissolution
On three real machines, the complete model averaged 72.3%.
∆ Avg. Changes to the relatively complete model ∆ percentage pointsIt's a great way to get started.
Real-time feedback is more valuable than predictions: removing only predictions in real time (63.0%(-9.3); removing only predictions when they remain (69.7%(-2.7);; the role of predictions is to "prepare in advance", the role of feedback is to "correct on the spot", the latter irreplaceable.
Changes in the generation process > Changes after generation: motion space residual 61.3%, and untouchable 61.7%) are almost the same.
The isolation mechanism cannot be avoided: simultaneously removing the attention mask and the stop gradient, it falls to 62.7%, close to the level of "no touch". This indicates that if not isolated, the large pre-training features of the joint optimization movement experts will directly overwhelm the touch signal.
"Invisible" and "Not Updated" are the prerequisites for tactile speech.
In addition, the removal of asymmetrical reasoning from the bottle cap from 67 to 56 indicates that intensive contact tasks do depend on high-frequency refreshments.
Written at the end
The value of the touch is not so much to look at the signal loop as to change its direction at the moment the loop is generated in the motion.
Well, that's the access position judgement given by this job.
Alongside this, it also provides a reverse instructional material:π0.5 plus touch is the opposite. The large visual-dominated model has a strong representational inertia, and if the new model is simply "seen" without a path that can actually act on the output, it is likely to be ignored and even become noise.
At the same time, TF2-Steering is still "100 instructions per task + post-training", and the universality of touch is also dependent on the visual mainstream.
Edited by Dr. Little Monkey
I'm not sure what I'm doing.
Ref
The paper is titled: TF2-Steering: Action Refinement with Tactile Forecasts and Feedback via Velocity-Field Steering.
Link to the project:https://yiming-cc.github.io/tf2-steering/
The following is a link to the article:
