Questions

1.Recall the definition of the interaction matrix and feature Jacobian.
Write an expression that involves the three potential control spaces of a robot.

Here is the definition of interaction matrix,which could tell us feature varieties when camera moves.
s ˙ = ( ∂ s ∂ r ) d r d t = J s T \dot{s}=\left(\frac{\partial s}{\partial r}\right) \frac{d r}{d t}=J_{s} T

s˙=(∂r∂s​)dtdr​=Js​T
where T is a camera velocity screw. V V V and Ω \Omega Ω represent its translational and rotational component respectively.
T = d r d t = ( V T , Ω T ) T T=\frac{d r}{d t}=\left(V^{T}, \Omega^{T}\right)^{T} T=dtdr​=(VT,ΩT)T
In our course, we represent the matrix as following.
s ˙ = L s v c \dot s=L_s\mathbf{v_c} s˙=Ls​vc​
Then we have to understand the definition of feature Jacobian.It tells how features change when robot moves.
s ˙ = J s q ˙ \dot s=J_s \dot \mathbf{q} s˙=Js​q˙​
First,we have Jacobian matrix between joint velocities and end-effector velocities.
b a s e v e = J q q ˙ ^{base}\mathbf{v_e}=J_q\dot \mathbf{q}

baseve​=Jq​q˙​
We have e v e = [ e R b 0 3 × 3 0 3 × 3 e R b ] b v e ^e\mathbf{v}_e=\left[\begin{array}{cc} ^e\mathbf{R}_{b} & \mathbf{0}_{3 \times 3}\\ \mathbf{0}_{3 \times 3} & { }^{e} \mathbf{R}_{b} \end{array}\right]{}^b\mathbf{v}_e eve​=[eRb​03×3​​03×3​eRb​​]bve​

c v c = [ c R e [ c t e ] × c R c 0 3 × 3 c R e ] e v e = c W e e v e ^c\mathbf{v}_c=\left[\begin{array}{cc} {}^c\mathbf{R}_{e} & {\left[{ }^{c} \mathbf{t}_{e}\right]_{\times}{ }^{c} \mathbf{R}_{c}} \\ \mathbf{0}_{3 \times 3} & { }^{c} \mathbf{R}_{e} \end{array}\right]{}^e\mathbf{v}_e={}^c\mathbf{W}_e{}^e\mathbf{v}_e cvc​=[cRe​03×3​​[cte​]×​cRc​cRe​​]eve​=cWe​eve​
We want to link velocities of the end-effector velocities and the camera velocities.Finally,we can obtain :
J s = L s c W e e R b J q J_s=L_s{}^c\mathbf{W}_e ^e\mathbf{R}_{b} J_q Js​=Ls​cWee​Rb​Jq​
THe size of the feature matrix is ( m × n ) (m \times n) (m×n), L s ( m × 6 ) L_s(m \times 6) Ls​(m×6), J q ( 6 × n ) J_q(6\times n) Jq​(6×n).

QUESTION:I am still confused about the following equation.Why ∂ s i ∂ t − ∂ s i ∗ ∂ t \frac{\partial \mathbf{s}_{i}}{\partial t}-\frac{\partial \mathbf{s}_{i}^{*}}{\partial t} ∂t∂si​​−∂t∂si∗​​ disappears and e ˙ i = s ˙ i \dot e_i=\dot s_i e˙i​=s˙i​
e ˙ i = ∂ s i ∂ q q ˙ + ∂ s i ∂ t − ∂ s i ∗ ∂ t = J i q ˙ = s ˙ i \dot{\mathbf{e}}_{i}=\frac{\partial \mathbf{s}_{i}}{\partial \mathbf{q}} \dot{\mathbf{q}}+\frac{\partial \mathbf{s}_{i}}{\partial t}-\frac{\partial \mathbf{s}_{i}^{*}}{\partial t}=\mathbf{J}_{i} \dot{\mathbf{q}}=\dot{\mathbf{s}}_{i} e˙i​=∂q∂si​​q˙​+∂t∂si​​−∂t∂si∗​​=Ji​q˙​=s˙i​

Feedback: As for the slide remark: ds/dt is 0 because we assume a motionless observed object. ds*/dt is also 0 because we assume a constant desired feature value s*.I’ll come back to it next lecture.

three potential control spaces of a robot

We have seen that the interaction matrices of numerous visual features have been derived in the past decades. What are the properties that are usually analyzed?

1. L(s) 3D behavior,more 0 in the matrix means we could have couple actions between feature spaces and operator space.(line moving in feature s means direct line in 3D position space.)
2. Singularities,3 points in a circle,when our camera moves in the cylinder(along the circle),we are in the singularity situation.
3. Local minimum

Why do we always assume we can only estimate the interaction matrix?

sensor noise and non-measurement parameters,3D shape model

A very coarse estimation of the interaction matrix is to use the value at the desired position.

1. Give the advantages and drawbacks of this approach.
2. Give a type of visual features for which this approach could be useful.

If we use the value at desired position,it means the image we have measured is the desired one.We could use (0,0) make our interaction matrix has more components equal to zero.

I would like to use cylindrical coordinates.I think in this estimation situation,there is large rotation error,polar induces a better trajectory.

which means our interaction matrix is const,we can use the current error to get velocity output v c = − λ L + ^ ( s − s ∗ ) v_c=-\lambda\hat{L^{+}}(s-s*) vc​=−λL+^(s−s∗),when s far from the desired value,we have a better L ^ \hat L L^.Drawback :coarse estimation leads to unstable control,negative eigenvalue increases errors.
This approach could be useful if we have much more features,in this way,our interaction matrix is too huge,we just want to calculate it once using s = s ∗ s=s^* s=s∗.

A visual servoing is performed from the observation of a 3D segment.

1. Write the two main modeling approaches that may be used in this case.

point,length of the segment

2. Which of the two can be considered preferable?

middle points

3. Try to have as many terms of the corresponding interaction matrix equal to 0, and give the assumption that is done to do so.

Any assumptions is OK.

Fish-eye camera

1. Assuming the camera is calibrated, compute the interaction matrix of the normalized point (x, y) = (X/η, Y /η).
   You may use η or ρ if they appear instead of reducing everything to X, Y, Z, α and β.

For question 1,as the definition of the interaction matrix,
Feature s = ( x , y ) s=(x,y) s=(x,y) in the normalized coordinate,we have ( x , y ) = ( X η , Y η ) (x,y)=(\frac{X}{\eta},\frac{Y}{\eta}) (x,y)=(ηX​,ηY​)
s ˙ = L 2 D v c \dot s =L_{2D}\mathbf{v_c} s˙=L2D​vc​
s ˙ = M X ˙ = [ ∂ x ∂ X ∂ x ∂ Y ∂ x ∂ Z ∂ y ∂ X ∂ y ∂ Y ∂ y ∂ Z ] X ˙ \dot s = \mathbf{M} \dot X=\left[\begin{array}{ccc} \frac{\partial x}{\partial X} & \frac{\partial x}{\partial Y}& \frac{\partial x}{\partial Z} \\ \frac{\partial y}{\partial X} & \frac{\partial y}{\partial Y}& \frac{\partial y}{\partial Z} \end{array}\right]\dot X s˙=MX˙=[∂X∂x​∂X∂y​​∂Y∂x​∂Y∂y​​∂Z∂x​∂Z∂y​​]X˙

[ ∂ x ∂ X ∂ x ∂ Y ∂ x ∂ Z ∂ y ∂ X ∂ y ∂ Y ∂ y ∂ Z ] = [ 1 η − X η 2 ∂ η ∂ X 0 − X η 2 [ 1 − α + α Z ρ ] 0 1 η − Y η 2 ∂ η ∂ Y − Y η 2 [ 1 − α + α Z ρ ] ] \left[\begin{array}{ccc} \frac{\partial x}{\partial X} & \frac{\partial x}{\partial Y}& \frac{\partial x}{\partial Z} \\ \frac{\partial y}{\partial X} & \frac{\partial y}{\partial Y}& \frac{\partial y}{\partial Z} \end{array}\right]=\left[\begin{array}{ccc} \frac{1}{\eta} -\frac{X}{\eta^2}\frac{\partial \eta}{\partial X} & 0&- \frac{X}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] \\ 0 &\frac{1}{\eta}-\frac{Y}{\eta^2}\frac{\partial \eta}{\partial Y} & -\frac{Y}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] \end{array}\right] [∂X∂x​∂X∂y​​∂Y∂x​∂Y∂y​​∂Z∂x​∂Z∂y​​]=[η1​−η2X​∂X∂η​0​0η1​−η2Y​∂Y∂η​​−η2X​[1−α+ραZ​]−η2Y​[1−α+ραZ​]​]
where
∂ η ∂ X = α 1 2 ρ 2 β X = 1 ρ α β X ∂ η ∂ Y = α 1 2 ρ 2 β Y = 1 ρ α β Y \frac{\partial \eta}{\partial X}=\alpha\frac{1}{2\rho}2\beta X=\frac{1}{\rho}\alpha\beta X\\ \frac{\partial \eta}{\partial Y}=\alpha\frac{1}{2\rho}2\beta Y=\frac{1}{\rho}\alpha\beta Y ∂X∂η​=α2ρ1​2βX=ρ1​αβX∂Y∂η​=α2ρ1​2βY=ρ1​αβY
Because of η = α ρ + ( 1 − α ) Z ρ = β ( X 2 + Y 2 ) + Z 2 \begin{array}{l} \eta=\alpha \rho+(1-\alpha) Z \\ \rho=\sqrt{\beta\left(X^{2}+Y^{2}\right)+Z^{2}} \end{array} η=αρ+(1−α)Zρ=β(X2+Y2)+Z2 ​​

Then we would like to calculate L 3 D L_{3D} L3D​.
X ˙ = L 3 D v c X ˙ = − v + [ X ] × ω = [ − l 3 [ X ] × ] v L 3 D = [ − l 3 [ X ] × ] = [ − 1 0 0 0 − Z Y 0 − 1 0 Z 0 − X 0 0 − 1 − Y X 0 ] L s = M L 3 D = [ 1 η − X η 2 ∂ η ∂ X 0 − X η 2 [ 1 − α + α Z ρ ] 0 1 η − Y η 2 ∂ η ∂ Y − Y η 2 [ 1 − α + α Z ρ ] ] [ − 1 0 0 0 − Z Y 0 − 1 0 Z 0 − X 0 0 − 1 − Y X 0 ] \dot X=L_{3D}\mathbf{v_c}\\ \dot{\mathbf{X}}=-\boldsymbol{v}+[\mathbf{X}]_{\times} \boldsymbol{\omega}=\left[\begin{array}{ll} -\mathbf{l}_{3} & {[\mathbf{X}]_{\times}} \end{array}\right] \mathbf{v}\\ L_{3D}=\left[\begin{array}{ll} -\mathbf{l}_{3} & {[\mathbf{X}]_{\times}} \end{array}\right]=\left[\begin{array}{llllll} -1&0&0&0&-Z&Y\\ 0&-1&0&Z&0&-X\\ 0&0&-1&-Y&X&0 \end{array}\right]\\ Ls= ML_{3D}=\left[\begin{array}{ccc} \frac{1}{\eta} -\frac{X}{\eta^2}\frac{\partial \eta}{\partial X} & 0&- \frac{X}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] \\ 0 &\frac{1}{\eta}-\frac{Y}{\eta^2}\frac{\partial \eta}{\partial Y} & -\frac{Y}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] \end{array}\right]\left[\begin{array}{llllll} -1&0&0&0&-Z&Y\\ 0&-1&0&Z&0&-X\\ 0&0&-1&-Y&X&0 \end{array}\right] X˙=L3D​vc​X˙=−v+[X]×​ω=[−l3​​[X]×​​]vL3D​=[−l3​​[X]×​​]=⎣⎡​−100​0−10​00−1​0Z−Y​−Z0X​Y−X0​⎦⎤​Ls=ML3D​=[η1​−η2X​∂X∂η​0​0η1​−η2Y​∂Y∂η​​−η2X​[1−α+ραZ​]−η2Y​[1−α+ραZ​]​]⎣⎡​−100​0−10​00−1​0Z−Y​−Z0X​Y−X0​⎦⎤​
L s = [ − 1 η + X η 2 ∂ η ∂ X 0 X η 2 [ 1 − α + α Z ρ ] X Y η 2 [ 1 − α + α Z ρ ] − Z η + X Z η 2 ∂ η ∂ X − X 2 η 2 [ 1 − α + α Z ρ ] Y η − X Y η 2 ∂ η ∂ X 0 1 η − Y η 2 ∂ η ∂ Y Y η 2 [ 1 − α + α Z ρ ] Z η − Y Z η 2 ∂ η ∂ Y + Y 2 η 2 [ 1 − α + α Z ρ ] − X Y η 2 [ 1 − α + α Z ρ ] − X η + X Y η 2 ∂ η ∂ Y ] L_s=\left[\begin{array}{ccc} -\frac{1}{\eta} +\frac{X}{\eta^2}\frac{\partial \eta}{\partial X} & 0& \frac{X}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] &\frac{XY}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}]&-\frac{Z}{\eta}+\frac{XZ}{\eta^2}\frac{\partial\eta}{\partial X}-\frac{X^2}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}]&\frac{Y}{\eta} -\frac{XY}{\eta^2}\frac{\partial \eta}{\partial X} \\ 0 &\frac{1}{\eta}-\frac{Y}{\eta^2}\frac{\partial \eta}{\partial Y} & \frac{Y}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}] &\frac{Z}{\eta}-\frac{YZ}{\eta^2}\frac{\partial\eta}{\partial Y}+\frac{Y^2}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}]&-\frac{XY}{\eta^2} [1-\alpha+\frac{\alpha Z}{\rho}]&-\frac{X}{\eta} +\frac{XY}{\eta^2}\frac{\partial \eta}{\partial Y} \end{array}\right] Ls​=[−η1​+η2X​∂X∂η​0​0η1​−η2Y​∂Y∂η​​η2X​[1−α+ραZ​]η2Y​[1−α+ραZ​]​η2XY​[1−α+ραZ​]ηZ​−η2YZ​∂Y∂η​+η2Y2​[1−α+ραZ​]​−ηZ​+η2XZ​∂X∂η​−η2X2​[1−α+ραZ​]−η2XY​[1−α+ραZ​]​ηY​−η2XY​∂X∂η​−ηX​+η2XY​∂Y∂η​​]
QUESTION:About the final representation of our interaction matrix L s L_s Ls​,could I remain L s ( X , Y , Z ) L_s(X,Y,Z) Ls​(X,Y,Z) where ( X , Y , Z ) (X,Y,Z) (X,Y,Z) are 3D point parameters in the 3D space? or I must change ( X , Y , Z ) (X,Y,Z) (X,Y,Z) into 2D point parameters ( x , y ) (x,y) (x,y) in the image frame.If it is a requirement that we must use L s ( x , y ) L_s(x,y) Ls​(x,y)

L s = [ − 1 η + x 2 ρ α β 0 x η [ 1 − α + α Z ρ ] x y [ 1 − α + α Z ρ ] − Z η + 1 ρ α β x 2 Z − x 2 [ 1 − α + α Z ρ ] y − 1 ρ α β x 2 y η 0 1 η − y 2 ρ α β y η [ 1 − α + α Z ρ ] Z η − 1 ρ α β y 2 Z + y 2 [ 1 − α + α Z ρ ] − x y [ 1 − α + α Z ρ ] − x + 1 ρ α β x y 2 η ] L_s=\left[\begin{array}{ccc} -\frac{1}{\eta} +\frac{x^2}{\rho}\alpha\beta& 0& \frac{x}{\eta} [1-\alpha+\frac{\alpha Z}{\rho}] &xy[1-\alpha+\frac{\alpha Z}{\rho}]&-\frac{Z}{\eta}+\frac{1}{\rho}\alpha\beta x^2Z-x^2 [1-\alpha+\frac{\alpha Z}{\rho}]&y -\frac{1}{\rho}\alpha\beta x^2y\eta\\ 0 &\frac{1}{\eta}-\frac{y^2}{\rho}\alpha\beta& \frac{y}{\eta} [1-\alpha+\frac{\alpha Z}{\rho}] &\frac{Z}{\eta}-\frac{1}{\rho}\alpha\beta y^2Z+y^2 [1-\alpha+\frac{\alpha Z}{\rho}]&-xy [1-\alpha+\frac{\alpha Z}{\rho}]&-x+\frac{1}{\rho}\alpha\beta xy^2\eta \end{array}\right] Ls​=[−η1​+ρx2​αβ0​0η1​−ρy2​αβ​ηx​[1−α+ραZ​]ηy​[1−α+ραZ​]​xy[1−α+ραZ​]ηZ​−ρ1​αβy2Z+y2[1−α+ραZ​]​−ηZ​+ρ1​αβx2Z−x2[1−α+ραZ​]−xy[1−α+ραZ​]​y−ρ1​αβx2yη−x+ρ1​αβxy2η​]

2. Give the values of (α, β) that have the model fall back to perspective projection.
   Show that in this case the interaction matrix is the classical 2D one.

After comparing with L c l a s s i c 2 D = [ − 1 / Z 0 x / Z x y − ( 1 + x 2 ) y 0 − 1 / Z y / Z 1 + y 2 − x y − x ] L_{classic 2D}=\left[\begin{array}{ccccc} -1 / Z & 0 & x / Z & x y & -\left(1+x^{2}\right) & y \\ 0 & -1 / Z & y / Z & 1+y^{2} & -x y & -x \end{array}\right] Lclassic2D​=[−1/Z0​0−1/Z​x/Zy/Z​xy1+y2​−(1+x2)−xy​y−x​]
We observed that α = 0 \alpha=0 α=0 make the interaction matrix into the classical 2D one

RGB-D Camera

We have a camera Z(x,y,t).
Z ˙ = ∂ Z ∂ x x ˙ + ∂ Z ∂ y y ˙ + ∂ Z ∂ t \dot\mathbf{Z}=\frac{\partial Z}{\partial x}\dot x+\frac{\partial Z}{\partial y}\dot y+\frac{\partial Z}{\partial t} Z˙=∂x∂Z​x˙+∂y∂Z​y˙​+∂t∂Z​
When we calculate the camera,we use ∂ Z ∂ t = Z ˙ − ∂ Z ∂ x x ˙ − ∂ Z ∂ y y ˙ \frac{\partial Z}{\partial t}=\dot\mathbf{Z}-\frac{\partial Z}{\partial x}\dot x-\frac{\partial Z}{\partial y}\dot y ∂t∂Z​=Z˙−∂x∂Z​x˙−∂y∂Z​y˙​ , L z = L z ˙ − [ A B ] L 2 D L_z=L_{\dot z}-[A \quad B]L_{2D} Lz​=Lz˙​−[AB]L2D​
Where L z ˙ L_{}\dot z L​z˙ is the 3rd line of L 3 D L_{3D} L3D​.
when observing a plane perpendicular to the Z-axis,A and B are 0.Our interaction matrix Rank is 3.